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Foreword 


As an international, nonprofit cooperative of national research institutions and 
governmental research agencies, the International Association for the Evaluation of 
Educational Achievement (IEA) has conducted more than 30 large-scale comparative 
studies in countries around the world. These studies have reported on educational 
policies, practices, and learning outcomes on a wide range of topics and subject matters. 
These investigations have proven to be a key resource for monitoring educational quality 
and progress within individual countries and across a broad international context. 

The International Computer and Information Literacy Study (ICILS) follows a series of 
earlier IEA studies that had, as their particular focus, information and communication 
technologies (ICT) in education. The first of these, the Computers in Education Study 
(COMPED), was carried out in 1989 and again in 1992 for the purpose of reporting on 
the educational use of computers in the context of emerging governmental initiatives 
to implement ICT in schools. The next series of projects in this area was the Second 
Information Technology in Education Study (SITES), conducted in 1998-1999 
(Module 1), 2001 (Module 2), and 2006. These projects provided an update on the 
implementation of computer technology resources in schools and their utilization in 
the teaching process. 

The continuing rapid development of computer and other information technologies 
has transformed the environment in which young people access, create, and share 
information. Many countries, having recognized the imperative of digital technology 
in all its forms, acknowledge the need to educate their citizens in the use of these 
technologies so that they and their society can secure the future economic and social 
benefits of proficiency in the use of digital technologies. Within this context, many 
questions relating to the efficacy of instructional programs and how instruction is 
progressed in the area of digital literacy arise. 

ICILS represents the first international comparative study to investigate how students 
are developing the set of knowledge, understanding, attitudes, dispositions, and skills 
that comprise computer and information literacy (CIL) in order to participate effectively 
in the digital age. The aim of ICILS is to report on student achievement by way of 
an authentic computer-based assessment. In order to help explain variations in CIL 
outcomes internationally and to inform policymakers on the possible contribution of 
education systems for digital CIL as an essential skill, ICILS will also capture information 
about the broader in- and out-of-school contexts in which student proficiency is 
developed. 

This publication, the ICILS Assessment Framework, describes the background, 
constructs, and design of the assessment. The framework development process — as in 
all IEA studies — was a highly collaborative effort that benefited from the contributions 
of a number of individuals and groups involved in the study. The project advisory 
committee (PAC) and study participants were instrumental in this effort. 
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International studies of the scale of ICILS require a significant financial commitment 
from IEA and its partners. In addition to IEA’s own resources, critical funding for 
the study has come from the participation fees of the ICILS countries and from the 
European Commission Directorate-General for Education and Culture’s grant to the 
European countries participating in the project. 

I would like to express my thanks to the team of researchers from the international study 
center located at the Australian Council for Educational Research (ACER), especially 
to research director Julian Fraillon, project coordinator John Ainley, and assessment 
coordinator Wolfram Schulz for their leadership. My special thanks also go to colleagues 
from the IEA Secretariat in Amsterdam, the Netherlands, and the IEA Data Processing 
and Research Center in Hamburg, Germany, for their support. I also extend thanks to 
the staff at SoNET Systems, Melbourne, Australia, involved in developing the software 
for the computer-based student assessment. Particular thanks go to SoNET’s Mike 
Janie and Stephen Birchall. I also acknowledge the work of Jean Dumais from Statistics 
Canada, who served as sampling referee. 

I express my sincere gratitude to the members of PAC for their thoughtful feedback 
on earlier versions of the assessment framework: John Ainley (ACER), Ola Erstad 
(University of Oslo), Kathleen Scalise (University of Oregon), and Alfons ten 
Brummelhuis (Kennisnet) . Kjartan Steffensen, Jesus Maria Aiquezar-Sabadie (European 
Commission), and the IEA Publications and Editorial Committee also contributed to 
the review of the framework, and Paula Wagemaker edited the document. 

ICILS would not be possible without the commitment of the national research 
coordinators from participating countries. They play a crucial role in the development 
and implementation of each IEA study by ensuring that it embodies the interests of the 
broader community of researchers, policymakers, and practitioners. 


HANS WAGEMAKER 

EXECUTIVE DIRECTOR, IEA 


Overview 


Purpose of the study 

The purpose of the International Computer and Information Literacy Study 2013 
(ICILS 2013) is to investigate, in a range of countries, the ways in which young people 
are developing computer and information literacy (CIL) to support their capacity 
to participate in the digital age. To achieve this aim, the study will assess student 
achievement through an authentic computer-based assessment of CIL administered 
to students in their eighth year of schooling. It will also collect and report on analyses 
of data about student use of computers and other digital devices as well as students’ 
attitudes toward the use of computers and other digital tools. 

Some of these data represent outcomes, and others represent aspects of computer use 
that inform an understanding of the broader context in which CIL is developed in 
young people. Further contextual information collected during implementation of the 
student assessment will include background data about the participating students and 
data from the participating teachers, schools, and education systems about the policies, 
resources, and pedagogies relating to how CIL is taught and learned. 

This study is the first of its kind — in terms of emphasis on students’ acquisition of 
CIL — in international comparative research. It is a response to the increasing use of 
information and communication technologies (ICT) in modern society and the need 
for citizens to develop relevant skills in order to participate effectively in the digital 
age. It also addresses the necessity for policymakers and education systems to have a 
better understanding of the contexts and outcomes of ICT-related education programs 
in their countries. 

The purpose of the ICILS framework is to articulate the basic structure of the study. It 
provides a description of the field and the constructs to be measured. It also outlines 
the design and content of the measurement instruments, sets down the rationale for 
those designs, and describes how measures generated by those instruments relate to the 
constructs. In addition, it hypothesizes relations between constructs so as to provide the 
foundation for some of the analyses that follow. Above all, the framework links ICILS to 
other work in the field. The contents of this assessment framework combine theory and 
practice in an explication of “both the ‘what’ and the ‘how’” (lago, 2009, p. 1) of ICILS. 

Background to the study 

The past two decades have witnessed the development and pervasive implementation 
of computer and other information technologies throughout societies around the 
world. There is consensus that the exchange and transformation of knowledge 
through information technologies is a feature of modern societies. Information 
technologies provide the tools for creating, collecting, storing, and using knowledge 
as well as for communication and collaboration (Kozma, 2003). The development of 
these technologies has changed not only the environment in which students develop 
skills for life but also the basis of many occupations and the ways in which various 
social transactions take place. Knowing about, understanding, and using information 
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technologies has become important for life in modern society, while assessment of these 
skills has become a component of monitoring student achievement in many education 
systems. This framework refers to this set of knowledge, understanding, and skills as 
CIL. 

Today, many countries recognize the importance that education and training in ICT 
has for providing citizens with the necessary skills to access information and participate 
in transactions through these technologies (Kozma, 2008). According to the United 
Kingdom’s Qualifications and Curriculum Authority (2007), ICT “is an essential skill 
for life and enables learners to participate in a rapidly changing world” (para. 1). The 
authors of a report on “E-learning Nordic,” a study that explored the impact of ICT 
on education in Nordic countries, observe that “ICT is seen as an essential cultural 
technique which can significantly improve the quality of education” (Pedersen et al., 
2006, p. 114). 

In 2008, under its i2010 strategy, the European Commission reported on 470 digital 
literacy initiatives in Europe and suggested that digital literacy is “increasingly becoming 
an essential life competence and the inability to access or use ICT has effectively become 
a barrier to social integration and personal development” (European Commission, 
2008, p. 4). The successor to the i20 10 strategy, the Digital Agenda for Europe, included 
“enhancing digital literacy, inclusion and skills” as one of seven priority areas for action 
(European Commission, 2013, para. 1) and led to the establishment of a conceptual 
framework for “benchmarking digital Europe” (European Commission, 2009a). In 
December 2011, the European Commission launched the DIGICOMP project, setting 
as its aims the following: 

- To identify the key components of Digital Competence in terms of the knowledge, 
skills and attitudes needed to be digitally competent; 

- To develop Digital Competence descriptors that will feed a conceptual framework/ 
guidelines that can be validated at European level, taking into account relevant 
frameworks currently available; 

- To propose a roadmap for the possible use and revision of a Digital Competence 
framework and descriptors of Digital Competence for all levels of learners. 
(Ferrari, 2012, p. 1, emphases original) 

Ferrari (2012) explains that digital competence (DC) is “both a requirement and a 
right of citizens, if they are to be functional in today’s society” (p. 3). She identifies, 
through an analysis of existing digital competence frameworks, seven competence 
areas: information management; collaboration; communication and sharing; creation 
of content and knowledge; ethics and responsibility; evaluation and problem solving; 
and technical operations. 

The United States has in place widespread and varied policies designed to encourage the 
use of ICT in schools (Anderson & Dexter, 2009) . In endeavoring to shape their curricula 
and assessments according to the policy directives, states have generally followed the 
National Educational Technology Standards established by the International Society 
for Technology in Education (2007). The US National Education Technology Plan 
implicitly and explicitly exhorts the development of skills that enable participation in 
the digital age. Goal 1.1 of the plan stresses that, regardless of the learning domain, 
“ . . . states should continue to consider the integration of 2 1 st-century competencies and 
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expertise, such as critical thinking, complex problem solving, collaboration, multimedia 
communication, and technological competencies demonstrated by professionals in 
various disciplines” (Office of Educational Technology, US Department of Education, 
2010, p. xvi). 

An assessment of technology competency (which includes ICT as one of three areas) is 
to be included in the US National Assessment of Educational Progress 2014 (WestEd, 
2010). The assessment covers proficiency with computers and software learning tools, 
networking systems and protocols, hand-held digital devices, and other technologies 
for accessing, creating, and communicating information and for facilitating creative 
expression. It also identifies five subareas of competency: construction and exchange of 
ideas and solutions, information research, investigation of problems, acknowledgement 
of ideas and information, and selection and use of digital tools (Institute of Education 
Sciences, National Center for Education Statistics, 2012). 

Over recent years, a number of countries in Latin America have increased their focus 
on the use of ICT in classrooms and also introduced one computer to every student in 
schools (commonly referred to as one-to-one resourcing) . Argentina, Brazil, Chile, Peru, 
and Uruguay are some of the countries that have implemented one-to-one computer 
policies (see, for example, Ministry of Education of the City of Buenos Aires, 2013; 
Ministry of Education of Uruguay, 2013; Severin & Capota, 2011; Severin, Santiago, 
Ibarraran, Thompson, & Cueto, 2011). 

Despite the international context wherein the importance of ICT-related literacies is 
universally acknowledged and widely regarded as increasing (Blurton, 1999; Kozma, 
2003), there is considerable variation among (and even within) countries with regard to 
explicit ICT curricula, resources, and teaching approaches (Educational Testing Service, 
2002; Kozma, 2008; OECD, 2005; Sturman & Sizmur, 2011). In addition to questions 
stemming from the variety of approaches in which ICT curricula are conceptualized 
and delivered, there are also questions about the nature of the role that schools and 
education systems play in supporting the development of ICT-related literacies among 
young people. 

In some countries, young people claim that they learn more about using computers out 
of school than they do in school (see, for example, Thomson & De Bortoli, 2007). Adults 
regard the new generation of young people as “digital natives” (Prensky, 2001) who 
have developed “sophisticated knowledge of and skills with information technologies” 
as well as learning styles that differ from those of previous generations (Bennett, Maton, 
& Kervin, 2008, p. 777). 

However, various commentators express concern about the value of labeling the new 
generation this way, and in particular challenge assumptions about the knowledge 
and skills that these digital natives acquire (see, for example, van den Beemt, 2010). 
In addition to identifying and discussing the “myths” associated with the notion of 
digital native, Koutropoulos (2011, p. 531) questions assumptions of homogeneity and 
pervasiveness, arguing that if we look “at the research ... we see that there is no one, 
monolithic group that we can point to and say that those are digital natives. As a matter 
of fact, the individuals who would fit the stereotype of the digital native appear to be in 
the minority of the population” (para 36, emphasis original). 

Questions are also being raised about the types of ICT use and consequent learning that 
young people experience, especially when they are away from school. Some scholars 
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query if young people are, indeed, developing through their ICT use the types of ICT- 
related knowledge, skills, and understandings that can be of significant value in later life. 
Crook (2008) characterizes the majority of young people’s communicative exchanges 
as “low bandwidth,” where the focus is on role allocation and co-operation rather than 
on genuine collaboration. Selwyn (2009) similarly challenges suppositions about the 
quality and value of much of young people’s self-directed ICT learning, observing that 
“if anything young people’s use of the internet can be described most accurately as 
involving the passive consumption of knowledge rather than the active creation of 
content” (p. 372). 

Today, the research community and policymakers continue to grapple with issues 
revolving around the development of digital literacies in young people. Although 
there is consistent rhetoric about the value of emergent digital literacies in providing 
positive life outcomes, just how school education can and should contribute to this 
process is less than clear. ICILS should bring greater clarity to these matters through its 
systematic investigation of CIL in young people and the ways in which CIL is developed. 
The development of a uniform research framework and an empirically based set of 
outcome standards is fundamental to any large-scale crossnational study such as ICILS. 
The framework and standards described in this document also serve as a means of 
informing and guiding ICT and CIL policy and thereby help to bring coherence to an 
area of learning that is of increasing international significance. 

The place of CIL in relation to traditional disciplines 

In some senses, CIL is analogous to reading literacy in that both are an end and a means 
in school education. At school, young people may learn to use ICT, and they may also 
use ICT to learn. Schools use ICT as a basis of instructional delivery systems designed 
to increase skills and knowledge in other learning areas. They also use ICT as a tool for 
accessing resources, communicating, analyzing, and conducting simulations. However, 
education systems also want students to develop ICT skills and knowledge and to 
understand the role of ICT in learning, work, and society. 

The use of ICT in schools for discipline-based and cross-disciplinary areas of 
instruction and for developing ICT-related skills and understandings has led to two 
approaches to measuring computer-based achievement. The first involves measuring 
area-specific achievement of computer use, such as online reading and solving 
mathematics and science-related problems. This method typically presupposes 
that ICT achievement is inseparable from subject-based achievement. The second 
approach — measuring ICT achievement as a discrete learning area — assumes that ICT 
achievement transcends individual disciplines and comprises a set of knowledge, skills, 
and understandings that learners can readily adapt and transfer to new contexts. 

In line with its broad aim of examining the outcomes of student CIL education across 
countries, ICILS has adopted the second approach. This approach also taps into the 
growing interest in the assessment of the ICT literacy- related competencies documented 
by Erstad (2006, 2010) and is consistent with the approach in other studies of ICT 
literacy such as the Assessment & Teaching of 21st Century Skills (Griffin, McGaw, & 
Care, 2012). 

The two primary justifications for researching CIL as a means of integrating and 
transcending individual learning areas are practical and conceptual. At the practical 
level, ICILS offers the opportunity to capture information on the outcomes of CIL 
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education without restricting CIL to the context of a single learning area. The integration 
of contexts within CIL provides both efficiency (by collecting data through a single 
study rather than multiple studies) and removes the effect of variations across studies 
(such as through population selections, timing of data collection, and variations across 
the role of CIL within learning areas) that may invalidate any attempts to compare CIL 
learning outcomes across learning areas. At the conceptual level, CIL-related skills are 
increasingly being regarded as a broad set of generalizable and transferable knowledge, 
skills, and understandings that individuals can use to manage the cross-disciplinary 
commodity that is information. 

The possibilities that CIL holds for integrating and processing information are seen 
to transcend the mere implementation and use of computer technologies within any 
single learning discipline (see, for example, Amtmann & Poindexter, 2008; Audunson 
& Nordlie, 2003; Educational Testing Service, 2002; Markauskaite, 2007). It is interest 
in these facets of CIL that sets ICILS apart from studies that focus solely on assessing 
online discipline- specific learning, such as online reading, writing, mathematics, and 
science. 

The critical conceptual difference between online assessments of discipline-specific 
learning areas and an assessment of CIL is that the latter measures students’ ability 
to use computers to manage and communicate information. In discipline-based 
assessments, the computer is used as a vehicle for students to express their discipline- 
specific knowledge, understanding, and skills. For example, assessments of online 
reading focus on students’ capacity to make sense of text by locating and interpreting 
information within the electronic text (see, for example, OECD, 2011). Items may focus 
on specific details, themes, main ideas, nuance, and authorial purpose and techniques 
evident in the text. The text is the primary information source and is understood to be 
deliberately crafted in order to communicate ideas. 

CIL-based assessments require students to search for and identify many possible 
information sources relating to a larger research question or proposition. Receptive 
tasks relating to students’ “reading” of a text in CIL focus on the likely trustworthiness 
and accuracy of the information and, by necessity, require students to be aware of the 
role of the text as a means of promoting an author’s potential agenda. CIL items do 
not necessitate the detailed reading of text that computer-based reading items do. In 
productive tasks, the capacity of students to “read” texts is evident only in the way in 
which they use the information in those texts to create new “information products”. 

Criteria relating to information use (that, by inference, relate to students’ capacity to 
make sense of computer-based texts) focus on the ways in which students select and 
then use (synthesize) the key ideas and information contained in the source texts. The 
point at which CIL transcends the conventional literacies and computer literacy that 
underpin it is when computer-based reading of texts combines with the necessity to 
synthesize and communicate information within a computer-based (hardware and 
software) context to a particular audience and for a particular purpose. 

The difference between CIL and computer-based reading is equally relevant to the 
learning areas of mathematics and science. In mathematics, for example, students 
frequently use computer technologies to draw graphs of data or functions or to rotate 
shapes in three-dimensional space. Items assessing computer-related mathematical 
literacy might require students to plot multiple functions for the purpose of determining 
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the points at which the graphs of the functions intersect (are “equal”) or to rotate a 
shape in space to determine how it appears from different perspectives. In these cases, 
students are using computer technology as a tool for demonstrating their understanding 
of underlying mathematical concepts. Similarly, students may be presented with a table 
of data and asked to draw a chart in order to include the data in a report. Assessment of 
this “tooF’-based level of computer use from a mathematics perspective would focus on 
the students’ ability to apply the necessary software commands to access the table and 
then draw the chart (or information product). 

However, when students’ work is assessed from a CIL perspective, how that work 
is presented in the information product, for example, with appropriate labeling 
conventions and sufficient text to explain the place and role of the data in it, would 
be considered. Thus, from the CIL perspective, the data or chart are information 
commodities that need to be used for a purpose rather than tools by which to express 
understanding of mathematical concepts. 

Computer technologies provide two immediate opportunities for computer-based 
science assessments. One is to use multimedia technology to demonstrate the physical 
manifestations of a scientific concept (such as color change or precipitation as evidence 
of a chemical reaction); the second is to provide software that students can use to help 
conduct investigations, run simulations, and generate data (see OECD, 2010a; Scalise 
et al., 2011). 

The first opportunity provides the advantage of demonstrating complex change without 
the need for text to describe that change, thereby reducing the reading load. The second 
opportunity provides a potentially efficient and safe way for students to complete 
scientific observations from which they can draw conclusions. In a computer-based 
science assessment, students may be asked, for example, to use simulation software to 
manipulate independent variables such as sunlight and water, and then to monitor the 
influence of both on the growth of plants. Here, students would be assessed on their 
capacity to determine a systematic way in which to manipulate the variables so that 
they could draw meaningful conclusions about the influence of water and light on plant 
growth. 

As was the case in the examples for reading and mathematics, the focus of science-based 
CIL is on students’ capacity to use scientific information as a commodity rather than as 
an expression of their understanding of specific science-related concepts. Thus, in the 
tool-based assessment relating to the effect of sun and water on plant growth, students 
would not be expected to express understanding of scientific reasoning through the 
manipulation of variables. Rather, they would be required to follow instructions about 
what to change (rather than deciding themselves what to change) and might be assessed 
on their capacity to use the software accurately. They might also be provided with 
unambiguous experimental data and required to make use of it as part of an integrated 
information product. 

Research questions, participants, and instruments 

ICILS aims to investigate the ways in which young people develop CIL to support 
their capacity to participate in the digital age. The key research questions for the study 
accordingly concern (1) the contexts in which CIL is developed, and (2) students’ 
proficiency in CIL. 
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Research questions 

1 ) What variations exist between countries, and within countries, in student computer 
and information literacy? 

2) What aspects of schools and education systems are related to student achievement 
in computer and information literacy with respect to: 

a) The general approach to computer and information literacy education; 

b) School and teaching practices regarding the use of technologies in computer 
and information literacy; 

c) Teacher attitudes to, and proficiency in, using computers; 

d) Access to ICT in schools; and 

e) Teacher professional development and within-school delivery of computer 
and information literacy programs? 

3) What characteristics of students’ levels of access to, familiarity with, and self- 
reported proficiency in using computers are related to student achievement in 
computer and information literacy? 

a) How do these characteristics differ among and within countries? 

b) To what extent do the strengths of the relations between these characteristics 
and measured computer and information literacy differ among countries? 

4) What aspects of students’ personal and social backgrounds (such as gender, 
socioeconomic background, and language background) are related to computer 
and information literacy? 

Participants 

The 1C1LS target population comprises students in their eighth year of schooling. In 
most education systems, this is Grade 8, provided that the average age of students in 
this grade is 13.5 years or above. In education systems where the average age in Grade 8 
is below 13.5, Grade 9 is defined as the ICILS target population. Schools with students 
enrolled in the target grade will be selected randomly proportional to size (PPS) . Within 
each sampled school, 20 students will be randomly selected from among all students 
enrolled in the target grade. 

The population for the ICILS teacher survey is defined as all teachers teaching regular 
school subjects to the target grade students at each sampled school. It includes only those 
teachers teaching the target grade students during the testing period and employed at 
school from the beginning of the school year. Fifteen teachers will be randomly selected 
from the teacher population at each sampled school. 

School-level data will be provided by the principal and ICT coordinator from each 
sampled school. 

Instruments 

The following instruments will be administered as part of ICILS. 

1) An international student test consisting of a computer-based set of authentic 
questions and tasks designed to measure student computer and information 
literacy (CIL). 

2) A student questionnaire consisting of a computer-based set of items measuring 
student background variables and access to, experience and use of, and familiarity 


14 


ICILS ASSESSMENT FRAMEWORK 


with ICT at home and at school. The questionnaire will also collect evidence of 
students’ attitudes toward using ICT. 

3) A teacher questionnaire, administered to selected teachers teaching any subject in 
the target grade. It will gather information about teacher background variables 
and use of ICT. The questionnaire includes items that ask teachers to rate their 
confidence in using computers in their teaching, to state their actual use of 
computers, and to express their attitudes toward using computers in teaching and 
learning. 

4) A school principal questionnaire, administered to the principals of sampled schools 
and designed to capture school characteristics, the application of ICT in teaching 
and learning, as well as aspects of the management of ICT at school. 

5) An ICT coordinator questionnaire, administered to ICT coordinators of sampled 
schools and designed to capture information on resources and support for ICT at 
schools. 

6) A national contexts survey, completed by experts in each country’s ICILS national 
research center. The survey will seek out information on the structure of the 
education system, the status of CIL- related education in the national curriculum 
and policies, initiatives and resourcing associated with ICT, and CIL-related 
education. The data obtained from this survey should provide a description of the 
contexts for CIL-related education in each country and assist interpretation of the 
results from the student, school, and teacher questionnaires. 


Computer and information literacy 
framework 


Overview 

The development and use of terms relating to CIL to describe a range of real-world 
proficiencies are widely documented in research literature. However, the development 
of context- specific constructs relating to CIL has led to a proliferation of frequently 
overlapping and confusing definitions (Bawden, 2001). 1 Paralleling this proliferation is 
the range of terms and definitions relating to media and critical literacies (Livingstone, 
Van Couvering, & Thumin, 2008). Similarly, definitions of computer literacy and digital 
literacy are numerous and overlapping (Tyner, 1998). 

Livingstone et al. (2008, p. 104) distinguish between scholarly preferences to achieve 
the following: 

introduce new terms to characterize these supposedly new skills (e.g., digital literacy, cyber 
literacy, Internet literacy and network literacy) ... [and those that] emphasize the continuities 
between old and new media and information and communication technologies by extending 
the term media literacy or literacy in general to encompass a converged concept of media and 
information literacies. 

Scholars who advocate for the latter position argue that the technological advances 
leading to the increasing range of media contents available on computers (video, audio 
streaming, and podcasts, for example) support the recognition and assimilation of ICT 
literacy-related constructs rather than the creation of new terms and constructs that 
purportedly access a “new” set of technical and cognitive competencies. 

One of the conceptual challenges for IC1LS has been to decide whether the definition 
and research construct of CIL should address a new set of competencies or emphasize 
its connection to existing ones. The decision eventually came down on the side of the 
second approach. Both definition and construct therefore derive from existing literature 
on computer-related and information-related literacies but their development took 
into account two fundamental parameters of ICILS: 

1) ICILS targets school-aged children (in their eighth year of school); and 

2) The assessment would be completed using computers and would focus on computer 
use. 

With these parameters in mind, the ICILS construct explicitly refers to computer 
literacy, rather than the broader contexts implicit (although not always measured in 
practice) in constructs relating to digital literacy, ICT literacy, and digital competence 
(Educational Testing Service, 2002; Janssen & Stoyanov, 2012; Ministerial Council for 
Education, Early Childhood Development, and Youth Affairs [MCEECDYA] , 2008). 


1 Virkus (2003), for example, lists terms used synonymously with information literacy. They include “‘infoliteracy’, 
‘informacy’, ‘information empowerment’, ‘information competence’, ‘information competency’, ‘information competencies’, 
‘information literacy skills’, ‘information literacy and skills’, ‘skills of information literacy’, ‘information literacy competence’, 
‘information literacy competencies’, ‘information competence skills’, ‘information handling skills’, ‘information problem 
solving’, ‘information problem solving skills’, ‘information fluency’, ‘information mediacy’ and ‘information mastery’...”. 
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In contrast to this practical consideration, the choice to focus on information rather 
than media literacy reflects a key difference that still exists between the two constructs. 

Both information and media literacy typically refer to the capacity to access, analyze, 
evaluate, and communicate information. What distinguishes the two is that the emphasis 
in media literacy is primarily on explicitly measuring “understanding” of information 
as an outcome, whereas the emphasis with respect to information literacy is primarily 
on the processes of information management (Catts 8c Lau, 2008; Christ 8c Potter, 1998; 
Livingstone et al., 2008; Ofcom, 2006; Peters, 2004). While it is a given that students 
must understand the information they are dealing with in order to evaluate and use it 
effectively, explicitly measuring that understanding is rarely the focus with information 
literacy. 

A second area of difference between media and information literacy is in their approach 
to the concept of information. Traditionally, media literacy has emphasized the range 
of forms in which information is “packaged,” whereas information literacy has focused 
on static texts (electronic or print). As indicated previously, advances in technology 
are increasingly blurring this distinction and may eventually render this difference 
redundant. 

Recently, the rapid development of web-based collaborative technologies along with 
improved internet connectivity in many countries has led to increasing research 
interest in skills associated with collaboration in learning contexts. Collaboration and 
collaborative problem-solving are included in conceptualizations informing projects 
such as Digital Competence (Ferrari, 2012) and 21st Century Skills (see, for example, 
Ananiadou 8c Claro, 2009; Partnership for 21st Century Skills, n. d.). Although these 
conceptualizations are regarded as outside the scope of ICILS, their enabling knowledge, 
skills, and understandings are not. 

ICILS was established to investigate the competencies associated with computer and 
information literacies as the enabling components of digital competence and 21st 
Century skills. Despite being developed independently of specific curriculum goals, 
ICILS does focus on what Lampe et al. (2010) consider should be technology-mediated 
educational priorities for middle school students. These include finding and synthesizing 
relevant resources, connecting to people and networks, and knowing how to present 
and express oneself online in general and through online systems in particular (p. 62). 

Defining computer and information literacy 

Information literacy constructs developed first through the fields of librarianship and 
psychology (Bawden, 2001; Church, 1999; Homann, 2003; Marcum, 2002) and are 
acknowledged as having the following processes in common: identifying information 
needs, searching for and locating information, and evaluating the quality of information 
(Catts 8c Lau, 2008; Livingstone et al., 2008; UNESCO, 2003). Most information literacy 
constructs (and particularly those developed over the past 10 years) extend these 
processes to include the ways in which the collected information can be transformed 
and used to communicate ideas (Catts 8c Lau, 2008; Peters, 2004). 

Computer literacy constructs in education typically focus not on the logical reasoning 
of programming (or the syntax of programming languages) but rather on declarative 
and procedural knowledge about computer use, familiarity with computers (including 
their uses), and, in some cases, attitudes toward computers (Richter, Naumann, 8c 
Groeben, 2000; Wilkinson, 2006). With digital technologies now serving as the world’s 
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primary information management resources, recent information literacy constructs 
have adopted and largely subsumed computer literacy constructs (see, for example, 
Cartelli, 2009). According to various commentators, the global concepts of “knowledge 
economy” and “information society” are the essential drivers of the ongoing integration 
of computer and information literacy. Catts and Lau (2008, p. 7), for example, make 
this observation: 

People can be information literate in the absence of ICT, but the volume and variable quality 
of digital information, and its role in knowledge societies, has highlighted the need for all 
people to achieve IL [information literacy] skills. For people to use IL within a knowledge 
society, both access to information and the capacity to use ICT are prerequisites. IL is, 
however, a distinct capacity and an integral aspect of adult competencies. 

The assumption that information is received, processed, and transmitted underlies CIL 
constructs. The key difference between explicit information literacy constructs (that 
still rely on and assume some computer proficiency) and computer literacy constructs 
appears to be that the latter allocates less importance than the former to the nature 
and constituent parts of the information processing that happens between reception 
and transmission. In essence, computer literacy focuses on a more direct path between 
reception and transmission than does information literacy, which emphasizes the 
processual steps involved as information is evaluated and transformed (Boekhorst, 2003; 
Catts & Lau, 2008). Over time, computer literacy and information literacy constructs 
have converged in the form of information and communication technologies (ICT) 
literacy and digital literacy. The following definitions of each show this convergence: 

• “ICT literacy is using digital technology, communications tools, and/or networks to 
access, manage, integrate, evaluate, and create information in order to function in a 
knowledge society” (Educational Testing Service, 2002, p. 2). 

• “ICT literacy is the ability of individuals to use ICT appropriately to access, manage 
and evaluate information, develop new understandings, and communicate with 
others in order to participate effectively in society” (Ministerial Council on Education, 
Employment, Training, and Youth Affairs [MCEETYA], 2005, p. 14). 

• Digital literacy is “the ability to use digital technology, communications tools, and / 
or networks to access, manage, integrate, evaluate, and create information in order to 
function in a knowledge society” (Lemke, 2003, p. 22). 

Common to these definitions is the assumption that individuals have the technical 
skills needed to use the technologies. All three definitions also list very similar sets of 
information literacy and communication processes. Each furthermore maintains that 
individuals need to acquire these forms of literacy in order to participate and function 
effectively in society. Binkley and colleagues’ (2012) documentation and synthesis of 
the operational definitions of ICT literacy that have developed over the past decade 
provide useful understanding not only of this development but also of how it taps into 
the construct of CIL used in ICILS. 

With reference to both the parameters of ICILS and the literature about ICT-related 
literacies, we use the following definition of CIL in this framework: 

Computer and information literacy refers to an individual’s ability to use computers 
to investigate, create, and communicate in order to participate effectively at home, 
at school, in the workplace, and in society. 
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Within the ambit of the ICILS assessment framework, then, this definition relies on 
and brings together technical competence (computer literacy) and intellectual capacity 
(conventional literacies including information literacy) to achieve a highly context- 
dependent communicative purpose that presupposes and transcends its constituent 
elements. This view of CIL is congruent with Audunson and Nordlie’s (2003 ) conceptual 
model of information literacy and is most closely aligned with the ICT literacy construct 
evident in the first of the three ICT and digital literacy definitions cited previously. 

Structure of the computer and information literacy construct 

The CIL construct includes the following elements: 

• Strand: This refers to the overarching conceptual category for framing the skills 
and knowledge addressed by the CIL instruments. 

• Aspect: This refers to the specific content category within a strand. 

The construct comprises two strands. One strand contains three aspects and the other 
strand contains four aspects (summarized in Figure 1 and described in detail below) . The 
aspects encompass the set of knowledge, skills, and understandings held in common by 
the range of definitions of ICT literacy and digital competency discussed previously and 
also by more recent ones from Ferrari (2012) and the Institute of Education Sciences, 
National Center for Education Statistics (2012). 

Figure 1: Conceptual structure of the CIL framework 


Computer and information literacy refers to an individual’s ability to use 
computers to investigate, create and communicate in order to participate 
effectively at home, at school, in the workplace, and in the community. 




Strand 1 

Collecting and managing 
information 


Strand 2 

Producing and exchanging 
information 


Aspect 1 .1 

Knowing about and understanding 
computer use 


Aspect 2.1 


Transforming information 

Aspect 2.2 

Creating information 

Aspect 2.3 

Sharing information 


Aspect 1 .2 

Accessing and evaluating information 

Aspect 1 .3 

Managing information 


Aspect 2.4 


Using information safely and securely 




CIL FRAMEWORK 


19 


The organization of aspects under strands does not presuppose an analytic structure 
with more than one subscale of CIL achievement. Rather, the two strands reflect the two 
primary uses of computers as receptive or productive CIL tools. Each aspect belongs to 
the strand with which it has the greatest, but not necessarily unique, congruence. 

Appendix B contains a progress map that details development of the CIL construct. 
Progress is described in terms of five levels for each of the two strands and includes the 
descriptors that were formulated on the basis of previous work in the field and supported 
in the data analyses from the international ICILS field trial. Each level includes a general 
description of CIL proficiency for that strand and level as well as specific examples of 
task completion deemed indicative of achievement at that level. 

Strands and aspects 

Strand 1: Collecting and managing information 

Collecting and managing information embraces the receptive and organizational 
elements of information processing and management, including the fundamental 
and generic skills and understandings associated with using computers. This strand 
comprises three aspects: 

• Knowing about and understanding computer use; 

• Accessing and evaluating information; 

• Managing information. 

Aspect 1.1: Knowing about and understanding computer use 

Knowing about and understanding computer use refers to a person’s declarative and 
procedural knowledge of the generic characteristics and functions of computers. This 
aspect focuses on the basic technical knowledge and skills that underpin our use of 
computers in order to work with information. Early constructs of ICT and digital literacies 
tended to omit this aspect, but this is no longer the case. For example, one of the seven 
areas of digital competency that Ferrari (2012) proposes is “technical operations,” while 
the Institute of Education Sciences, National Center for Education Statistics (2012), 
which administers the US’s National Assessment of Educational Progress, suggests that 
“although students are not expected to understand the inner workings of these devices, 
they should have enough of an understanding of the principles underlying them to 
appreciate the basics of how they work” (para. 4). 

At a declarative level, a person may know that computers use processors and memory 
to run programs, or that operating systems, wordprocessors, games, and viruses are 
examples of programs. They may be able to demonstrate knowledge that computers 
can be connected to and so can “communicate” with one another through networks, 
and that these can be local or global. They also may understand that the internet is a 
form of computer network which is run through computers and that websites, blogs, 
wikis, and all forms of computer software are designed to meet specific purposes. 

Procedural knowledge includes knowledge of the software interface conventions that 
help computer-users make sense of and operate unfamiliar software that adheres to 
these known interface conventions. Accordingly, at the procedural level, a person may 
know how to execute basic generic file and software functions such as opening and 
saving files in given locations, resizing images, copying and pasting text, and identifying 
file types by their extensions. The procedural knowledge included in Aspect 1.1 is thus 
limited to basic generic commands that are common across software environments. 
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Aspect 1.2: Accessing and evaluating information 

Accessing and evaluating information refers to the investigative processes that enable 
a person to find, retrieve, and make judgments about the relevance, integrity, and 
usefulness of computer-based information. The proliferation of information sources 
that use the internet as a communication medium means that users are required to 
filter the vast array of information to which they gain access before they can make use of 
it. However, the process of filtering in combination with the increasing intuitiveness of 
computer-based information search programs 2 is producing an ever greater integration 
of the processes of accessing and evaluating information. For this reason, accessing 
information and evaluating information are regarded as sufficiently integrated to 
warrant their inclusion as a single aspect, rather than separate aspects, of the CIL 
construct. 

The importance of accessing and evaluating information is also a direct result of the 
increasing quantity and range of available unfiltered computer-based (and delivered) 
information. Computer-based information is not only increasing in volume, but also 
constantly changing. While accessing information and evaluating information are 
rooted in conventional literacies, the dynamic multimedia and multimodal nature of 
computer-based information means that the processes of accessing and evaluating that 
contribute to the CIL construct are different from those that relate only to conventional 
literacies. The dynamic context of computer-based information therefore necessitates 
the use of an amalgam of a range of skills (i.e., those typically associated with digital 
and media literacies) that differ from and are broader than the range employed with 
conventional literacies. 

Examples of tasks that provide evidence of an individual’s ability to access and evaluate 
computer-based information include the following: 

• Selecting information from within a website or file list that is relevant to a particular 
topic; 

• Describing and explaining the functions and parameters of different computer-based 
information search programs; 

• Suggesting strategies for searching for information and/or adjusting the parameters 
of searches to target information better; 

• Recognizing and explaining characteristics of computer-based information (such as 
hyperbole and unsubstantiated claims) that detract from its credibility; 

• Recognizing that published information can have a hidden agenda; and 

• Suggesting and implementing strategies to verify the veracity of information (such as 
cross-checking information from multiple sources). 

Aspect 1.3: Managing information 

Managing information refers to the capacity of individuals to work with computer- 
based information. The process includes ability to adopt and adapt information 
classification and organization schemes in order to arrange and store information so 
that it can be used or reused efficiently. Managing information differs from Aspect 1.1 
(knowing and understanding computers) because it relates to making decisions about 
the way information is used, rather than to simply knowing or demonstrating that it 
can be used. 


2 These include search engines that tailor search results to individual searchers based on location, previous search behavior, 
and even the internet-use behavior of “friends” in a social network. 
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Managing information also differs from Aspect 1.2 (accessing and evaluating 
information) in that it relates to the ability to manage information in an environment 
where users have some control over the organization and structure of the information. 
Hence, for example, searching for a file that exists within a constrained file structure 
involves managing rather than accessing and evaluating information because it requires 
users to work with the known attributes of the closed file system rather than determine 
the likely searchable properties of an information product that would enable them to 
focus their search. 

Computer and information literacy, as manifest in this aspect of the domain, thus refers 
to the ability to manage information by, for example: 

• Creating a file structure in a directory according to given parameters; 

• Sorting or filtering information on an internet database; and 

• Recognizing the most efficient data structure for a given purpose within a simple 
database. 

Strand 2: Producing and exchanging information 

This strand, which focuses on using computers as productive tools for thinking, creating, 
and communicating, has four aspects: 

• Transforming information; 

• Creating information; 

• Sharing information; 

• Using information safely and securely. 

Aspect 2.1: Transforming information 

Transforming information refers to a person’s ability to use computers to change how 
information is presented so that it is clearer for specific audiences and purposes. This 
process typically involves using the formatting, graphics, and multimedia potential of 
computers to enhance the communicative effect or efficacy of (frequently text-based or 
numerical) information. 

The CIL manifest in this aspect of the domain refers to the ability to transform 
information by, for example: 

• Reformatting the titles in a document or presentation so as to enhance the flow of 
information; 

• Using, modifying, or creating images to supplement or replace text in a document 
(such as with a flow chart or diagram); 

• Creating a chart to represent a table of data; 

• Transferring data (such as temperature or velocity data) from a data-logger and 
displaying it in ways that illustrate patterns of change; and 

• Creating a short animated sequence of images to illustrate a sequence of events. 
Aspect 2.2: Creating information 

Creating information refers to a person’s ability to use computers to design and generate 
information products for specified purposes and audiences. These original products 
may be entirely new or may build upon a given set of information to generate new 
understandings. 
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The CIL manifest in this aspect of the domain therefore refers to the ability to create 
information by, for example: 

• Using a simple graphics program to design a birthday card; 

• Designing and writing a presentation that explains the key elements of an historical 
event; and 

• Using a given set of information to make recommendations in a report that integrates 
text, data, and graphics. 

Typically, the quality of information creation relates to how the information content is 
structured (whether or not the flow of ideas is logical and easy to understand) and the 
way in which layout and design features (such as images and formatting) are used to 
support understanding of the information produced. Even though information design 
and layout design are executed together in an information product, they are typically 
conceptualized and assessed as discrete elements of creating information. 

Aspect 2.3: Sharing information 

Sharing information refers to a person’s understanding of how computers are and can 
be used, as well as his or her ability to use computers to communicate and exchange 
information with others. Sharing information focuses on a person’s knowledge and 
understanding of a range of computer-based communication platforms, such as 
email, wikis, blogs, instant messaging, sharing media, and social networking websites. 
Given the rapidly changing nature of this area, Aspect 2.3 focuses on knowledge and 
understanding of information-based social conventions and, at the higher end of the 
achievement spectrum, the social impact of sharing information through computer- 
based communication media. 

The CIL manifest in this aspect of the domain accordingly refers to a person’s ability to 
share information by, for example: 

• Recognizing some key differences between computer-based communication media; 

• Using software to disseminate information (such as attaching a file to an email or 
adding or editing an entry in a wiki); 

• Evaluating the appropriateness of information in a given context; 

• Evaluating the best communication platform for a particular communicative 
purpose; and 

• Creating or modifying information products to suit a specified audience or 
purpose. 

Aspect 2.4: Using information safely and securely 

Using information safely and securely refers to a person’s understanding of the legal 
and ethical issues of computer-based communication from the perspectives of both 
the publisher and the consumer. Internet-based communication platforms increasingly 
are providing the facility for users to share information. With this facility comes the 
potential for misuse, particularly when dealing with personal information. Using 
information safely and securely also includes risk identification and prevention as well 
as the parameters of appropriate conduct. It furthermore focuses on the responsibility 
of users to maintain a certain level of technical computer security, such as using strong 
passwords, keeping virus software up to date, and not submitting private information 
to unknown publishers. 
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Current issues relating to this aspect include but are not limited to: 

• Identity theft; 

• Unauthorized access and impersonation; 

• Identity concealment; 

• Phishing; 

• Malicious software distribution; 

• Automatic collection of internet usage data; 

• Provision and use of personal information; and 

• Copyright restrictions for various media published on the internet. 

This aspect can be applied to both strands of the CIL construct depending on whether 
it is demonstrated in a receptive or a productive context. ICILS places it in Strand 2 
because, within the ICILS assessment context, safe and secure use is most frequently 
demonstrated when users actively produce or exchange information. 

The CIL manifest in this aspect of the domain relates to the proficiency to, for 
example: 

• Identify the strongest of a given set of passwords; 

• Explain the consequence (s) of making personal information publicly available; 

• Suggest ways to protect private information; 

• Understand how internet advertising targets users; and 

• Explain the techniques used in a phishing email scam. 



Contextual framework 


Overview 

This section describes the contextual information collected during ICILS in order to 
aid understanding of variation in students’ computer and information literacy (CIL). 
We provide a classification of factors that accords with the multilevel structure inherent 
in the process of student learning, and consider the relationship of these factors to the 
learning process (antecedents or processes). We also list the different kinds of variables 
that will be collected via the different ICILS contextual instruments and briefly outline 
prior findings from educational research in order to explain why these variables are 
included in ICILS. 

Classification of contextual factors 

When studying student outcomes related to CIL, it is important to set these in the 
context of the different factors influencing them. Students acquire competencies in 
this area through a variety of activities and experiences at the different levels of their 
education and through different processes in school and out of school. It is also likely, 
as Ainley, Enger, and Searle (2009) argue, that students’ out-of-school experiences 
of using ICT influence their learning approaches in school. Contextual variables can 
also be classified according to their measurement characteristics, namely, factual (e.g., 
age), attitudinal (e.g., enjoyment of computer use), and behavioral (e.g., frequency of 
computer use). 

Different conceptual frameworks for analyzing educational outcomes frequently point 
out the multilevel structure inherent in the processes that influence student learning 
(see, for example, Scheerens, 1990; Scheerens & Bosker, 1997; Schulz, Fraillon, Ainley, 
Losito, & Kerr, 2008; Travers, Garden, & Rosier, 1989; and Travers 8c Westbury, 1989). 
The learning of individual students is set in the overlapping contexts of school learning 
and out-of-school learning, both of which are embedded in the context of the wider 
community that comprises local, national, supranational, and international contexts. 
The contextual framework of ICILS therefore distinguishes the following levels: 

• The individual: This context includes the characteristics of the learner, the processes 
of learning, and the learner’s level of CIL. 

• Home environment: This context relates to a student’s background characteristics, 
especially in terms of the learning processes associated with family, home, and other 
immediate out-of-school contexts. 

• Schools and classroom: This context encompasses all school-related factors. Given 
the crosscurricular nature of CIL learning, it is not useful to distinguish between 
classroom level and school level. 

• Wider community: This level describes the wider context in which CIL learning 
takes place. It comprises local community contexts (e.g., remoteness and access to 
internet facilities) as well as characteristics of the education system and country. It 
also encompasses the global context, a factor widely enhanced by access to the world 
wide web. 
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The status of contextual factors within the learning process is also important. Factors 
can be classified either as antecedents or processes. 

• Antecedents are exogenous factors that condition the ways in which CIL learning 
takes place. They are contextual factors that are not directly influenced by learning- 
process variables or outcomes. It is important to recognize that antecedent variables 
are level-specific and may be influenced by antecedents and processes found at 
higher levels. Variables such as the socioeconomic status of the student’s family and 
the school intake along with home resources fall into this category. 

• Processes are those factors that directly influence CIL learning. They are constrained 
by antecedent factors and factors found at higher levels. This category contains 
variables such as opportunities for CIL learning during class, teacher attitudes toward 
using ICT for study tasks, and students’ use of computers at home. 

Both antecedents and processes need to be taken into account when explaining 
variation in CIL learning outcomes. Whereas antecedent factors shape and constrain 
the development of CIL, process factors can be influenced by the level of (existing) CIL 
learning. For example, the level and scope of classroom exercises using ICT generally 
depend on the existing CIL- related proficiency of the students. 

Figure 2 illustrates the basic classification of antecedent and process-related contextual 
factors in their relationship with CIL outcomes located at the different levels. Each 
type of factor at each level is accompanied by examples of variables that have the 
potential to influence learning processes and outcomes. The double-headed arrow in 
the figure between the process-related factors and outcome emphasizes the possibility 
of feedback between learning process and learning outcome. The single-headed arrow 
between antecedents and processes, in turn, indicates the assumption within the ICILS 
contextual framework of a unidirectional association at each contextual level. 

Figure 2: Contexts for CIL learning and learning outcomes 
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Reference to this general conceptual framework enables us to locate potential contextual 
factors on a two-by-four grid where antecedents and processes constitute the columns 
and the four levels the rows. Table 1 shows examples in each of these cells of the 
contextual variables collected by the ICILS instruments. The student questionnaire will 
collect data on contextual factors pertaining to the level of the individual student and his 
or her home context. The teacher, school principal, and ICT coordinator questionnaires 
are designed to locate contextual factors associated with the school/classroom level, 
while the national contexts survey and other available sources (e.g., published statistics) 
will gather contextual data at the level of the wider community. 


Table 1: Mapping of ICILS context variables to framework grid 


Level of ... 

Antecedents 

Processes 

Wider 

NCS & other sources: 

NCS & other sources: 

community 

Structure of education 
Accessibilty of ICT 

Role of ICT in curriculum 

School/classroom 

PrQ, ICQ, & TQ: 

PrQ, ICQ, & TQ: 


School characteristics 
ICT resources 

ICT use in teaching 

Student 

StQ: 

StQ: 


Gender 

ICT activities 


Age 

Use of ICT 

Home environment 

StQ: 

StQ: 


Parent SES 
ICT resources 

Learning about ICT at home 


Key: NCS = national contexts survey; PrQ = principal questionnaire; ICQ = ICT coordinator questionnaire; TQ 
= teacher questionnaire; StQ = student questionnaire. 


Contextual levels and variables 

The wider-community context 

The different levels of this context all have the potential to affect student learning at 
school or at home. Conceptually, this context has several levels: 

• Local communities, where remoteness and lack of stable and fast internet connections 
may affect conditions for ICT use; 

• Regional and national contexts, where communication infrastructure, educational 
structures, curricula, and general economic/social factors may be of importance; 
and 

• Supranational or even international contexts, where a long-term perspective brings 
in, for example, factors such as the general advance of ICT globally. 

The most important factors potentially explaining variation in CIL are located at the 
national level (or subnational level in those instances of subregions participating in the 
study). There is evidence of broad differences in terms of access to digital technology 
among countries across Europe as well as more broadly across the world (Korte & 
Husig, 2006; OECD, 2005). There is also evidence that, in recent years, a number of 
countries have invested in ICT infrastructure that will increase access to broadband 
internet within schools and homes or increase hardware access in schools. These 
countries include Australia, Canada, Estonia, Israel, New Zealand, Portugal, and the 
Republic of Korea (Bakia, Murphy, Anderson, & Trinidad, 2011). 
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Variables that describe the contexts of education systems will be collected from 
published sources as well as through the ICILS national contexts survey. Typically, 
published sources provide information about antecedent country-context variables 
while the national contexts survey will deliver data on antecedent and process variables 
at the level of and with respect to the education system. 

More specifically, the national contexts survey is designed to collect systemic data on 
the following: 

• Education policy and practice in CIL education (including curriculum approaches 
to CIL); 

• Policies and practices for developing the CIL expertise of teachers; and 

• Current debates on and reforms to the implementation of digital technology in 
schools (including approaches to the assessment of CIL and the provision of ICT 
resources in schools). 

Data pertaining to factors such as the structure of education systems and national 
curriculum orientations will also be captured so that they can be taken into account 
during interpretation of ICILS results. 

Antecedent variables at the level of the wider community 

International comparative research shows relatively strong associations between the 
general socioeconomic development of countries and student learning outcomes. 
ICILS will therefore select national (and where appropriate possible subnational) 
indicators of general human development status regularly reported by the United 
Nations Development Programme (UNDP, 2009). Examples of these indicators are 
gross domestic product per capita, access to education, and health statistics. 

Given ICILS’ focus on students’ CIL, it is important to take into account the general 
availability of and infrastructure for ICT. To this end, ICILS will collect, with the aim 
of describing the general ICT-related resources at the national level, information from 
published sources relating to variables such as the number of internet hosts. 

One example of a published source of data regarding national contexts is the ICT 
Development Index (IDI), developed by the International Telecommunications Union 
(2012). The IDI combines 11 indicators into a single measure that can be used as an 
index of ICT development for 154 countries or used as separate indicators. Another is 
the Networked Readiness Index (see, for example, Dutta & Mia, 2011). 

Data from a range of international surveys show that the provision of ICT resources 
in schools varies widely across countries (see, for example, Anderson & Ainley, 2010; 
Pelgrum & Doornekamp, 2009). In order to obtain information related to the general 
ICT resourcing of schools, the ICILS national contexts survey will collect data on 
school-based ICT infrastructure, hardware, and software, as well as policy expectations 
regarding these provisions. 

These data comprise system-level variables such as the number of computers per 
student, computers per teacher, internet connectivity (coverage and speed), software 
licensing arrangements, and the availability of digital curriculum resources. Analysis of 
this information will support evaluation of the premise that students in those schools 
with the highest levels of digital resourcing will have greater experience of and access to 
the use of CIL and consequently develop higher levels of CIL. 
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The national contexts survey will also gather data about a range of other characteristics 
of the education systems participating in ICILS. System-level variables related to this 
aspect include length of schooling, age-grade profiles, educational finance, and the 
structure of school education (e.g., study programs, public/private management), as 
well as the autonomy of educational providers. 

Process-related variables 

The process-related variables on CIL-related education policy that will be collected by 
the national contexts survey include the following: 

• The definition of and the priority that each country gives to CIL education in its 
educational policy and provision; 

• The name of and national or official definition given to CIL education; 

• The place of CIL education in educational reforms; 

• The main aims and goals of CIL education; and 

• The influence of different institutions or groups on decisions relating to those goals 
and aims. 

Because the ICILS contextual framework references policies and practices developed as 
outcomes of earlier large-scale surveys of ICT in education, ICILS also takes into account 
the process-related data in these studies’ reports and databases. The studies include IEA’s 
Second Information Technology in Education Study (SITES) (Plomp, Anderson, Law, & 
Quale, 2009), the European Commission’s Indicators of ICT in Primary and Secondary 
Education (European Commission, 2009b), and the International Experiences with 
Technology in Education survey, which covered policies and experiences in 2 1 countries 
(Bakia et al., 2011). 

The information from these studies shows that countries take different approaches 
to the implementation of CIL education in their curricula. Some education systems 
include it as a subject within the curriculum, whereas others include it by integrating it 
into other subjects. The explicitness with which countries describe their CIL curricula 
and the learning outcomes they want from them also vary across education systems. 
Some have very explicit curricula regarding CIL education and its expected learning 
outcomes; others describe CIL education as an “implicit” curriculum that weaves 
through the curriculum documents for other learning areas. 

In order to build on what is already known, the national contexts survey will gather 
data on the inclusion of CIL education (as a separate subject, integrated into different 
subjects, or as a crosscurricular approach) in the formal curriculum at different stages 
of schooling and in different study programs. It will also capture the nomenclature for 
CIL-related curriculum subjects and whether they are compulsory or optional in each 
program of study. Specific questions regarding the target grade in terms of curriculum 
emphasis and the amount of instructional time given to CIL education will also be 
asked. 

Another important process-related variable at the system level is the development of 
teacher expertise in CIL (Charalambos & Glass, 2007; Law, Pelgrum, & Plomp, 2008). 
Teacher education programs often provide aspiring teachers with opportunities 
to develop CIL-related competencies. To aid assessment of the variety of different 
approaches to teacher education in the field, the national contexts survey will gather 
(where applicable) data on CIL requirements for becoming a teacher, licensing or 
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certification procedures for teachers, and the backgrounds of CIL teachers. The survey 
will also seek out information on the extent to which CIL education is part of preservice 
or initial teacher education, on the availability of inservice or continuing professional 
development for CIL education, on the providers of these activities, and on expectations 
for teachers’ ongoing learning about developments in CIL education. 

Over the past few decades, many countries have undertaken educational reforms 
involving the introduction of digital technology. 3 A key feature of most national plans 
over the most recent decade is that they aspire to use ICT to transform patterns of 
learning and teaching and to develop capabilities useful within modern economies, 
rather than simply improve existing practice. However, countries differ in the extent 
to which they have introduced, or are introducing, digital technology into school 
education. There is also considerable variation in the level of priority given to this 
development, including the development of curriculum resources in the form of digital 
learning objects. The same can be said with respect to whether and how education 
systems assess CIL and whether they use ICT to assess other disciplines. The national 
contexts survey will therefore gather data about the priorities accorded to these digital 
developments and the nature of the debates surrounding them. 

School/classroom context 

Any study of students’ acquisition of CIL must acknowledge the key role that school 
and classroom contexts play in that acquisition. Use of ICT is increasingly becoming 
standard practice in education and is therefore an important part of preparing young 
people for participation in modern society. Factors associated with the school and 
classroom context will be collected through the teacher, school principal, and ICT 
coordinator questionnaires. In addition, the student questionnaire includes some 
questions gauging student perceptions about classroom practices related to ICT. Even 
though ICILS will not attempt to investigate the relationship between ICT use in 
schools or classrooms and achievement in academic learning areas such as language, 
mathematics, or science, it is of interest to note the suggestion of positive associations 
with achievement evident in the results of a recent meta-analysis conducted by Tamin, 
Bernard, Borokhovski, Abrami, and Schmid (2011). 

Antecedent variables at the school/classroom level 

In line with the need to take basic school characteristics into account when investigating 
variations in CIL, the questionnaire given to each school principal will collect 
information on student enrolment, teachers, the range of grades, and the location 
of each participating school. It will also collect data on school (public or private) 
management. Because, as noted earlier, we can regard ICT-related resources at school 
as a key context factor when studying students’ CIL, the school principal questionnaire 
will furthermore ask who, in the school, assumes responsibilities for the acquisition of 
ICT resources. 

The ICILS questionnaire for each school’s ICT coordinator includes questions on the 
availability of school-owned computing devices at school, their location within the 
school, how many students have access to them, which computer operating system 
the school mainly uses, and the number of years the school has been using ICT. The 


3 Two national plans for ICT in education are those of the United States (U.S. Department of Education, 2010) and Portugal 
(Ministry of Education of Portugal’s Plano Technologico Educagao 2007, cited in Bakia, et al., 201 1). 
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instrument will also collect data on the support the school provides for ICT use in 
teaching and learning in terms of personnel and technology or software resources. 
It additionally includes a question measuring the coordinator’s perceptions of the 
adequacy of the ICT on hand for learning and teaching at school. 

The background and experiences of teaching staff have the potential to influence the 
acquisition of student CIL. Results from SITES 2006 indicated that teachers are more 
likely to use ICT in their teaching when they have higher levels of self-confidence in 
using ICT in general (Law et al., 2008). SITES 2006 also indicated that, in most of 
the participating countries, ICT is more frequently used in science teaching than in 
mathematics teaching. 

The ICILS teacher questionnaire will therefore collect information on the general 
professional background of teaching staff (such as age, gender, subject taught at school) 
and on their ICT experience (number of years using ICT for teaching purposes, general 
use of computers at different locations, participation in ICT-related professional 
development activities, and perceived self-confidence in using ICT for different tasks). 
Teachers will also be asked to give their views on the positive and negative consequences 
of using ICT for teaching and learning, and to identify any factors that they think 
impede the use of ICT for teaching and learning at their school. 

SITES 2006 findings suggest that ICT use by science and mathematics teachers is 
influenced by the school principal’s views about its value, as well as the ICT-related 
support teachers have at hand (Law et al., 2008). Findings also indicate that ICT-related 
teaching and learning can be constrained or facilitated by the school’s stated curriculum 
and its policies with regard to ICT. The ICILS school principal questionnaire will 
therefore collect data on the following factors: 

• The extent to which the school has policies and procedures relating to ICT use; 

• The extent to which the school prioritizes ICT acquisition and resourcing; 

• Perception of the importance ascribed to ICT use in teaching at the school; 

• School-level expectations for teachers’ knowledge of and skills in using ICT; and 

• The extent to which teachers participate in ICT-related professional development. 

Process-related variables at the school/classroom level 

The emergence of ICT in school education has been seen for some time as having the 
potential to influence teaching and learning processes by enabling wider access to a 
range of resources, allowing greater power to analyze and transform information, and 
providing enhanced capacities to present information in different forms. However, the 
evolution of greater interactivity in more recent technologies (sometimes referred to as 
Web 2.0) has expanded these possibilities considerably (Greenhow, Robelia, & Hughes, 
2009). These developments have led to claims by some scholars that it is now possible 
for students to participate in extended projects that help them develop sophisticated 
concepts and skills through the use of simulation and visualization tools (Dede, 
2007). Commentators also argue that students can collaborate in developing learning 
experiences, generating knowledge, and sharing perspectives on experiences with other 
students. 

The aforementioned large-scale crossnational studies also show that schools and 
classrooms vary in the extent to which teachers use ICT in teaching. Burbules (2007) 
argues that although e-learning technologies have the potential to bring transformative 
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effects to classrooms, their implementation has been, for various reasons, surprisingly 
limited (see also Cuban, 2001). The ICILS teacher questionnaire accordingly asks 
teachers to consider one of their classes (specified in the questionnaire) and to identify 
(where applicable) the types of ICT applications used in that class, the type of and extent 
to which ICT is used as part of teaching practices and for particular learning activities 
in that class, and the emphasis placed on developing ICT-based student capabilities. 
The questionnaire also asks teachers about their perceptions of whether and how ICT is 
used as part of collaborative teaching and learning at their school. 

Actual student use of ICT in the learning process is another important factor. A segment 
of the teacher questionnaire therefore asks teachers to report on student involvement 
in different learning activities involving ICT use. The student questionnaire also asks 
students to report on how often they use computers at school, their use of computers 
for different school-related purposes, and the frequency with which they use ICT in 
their learning of different subjects. 

Home context 

Antecedent variables related to the home environment 

The influence of student home background on students’ acquisition of knowledge has 
been shown in many studies, and there is evidence that home background also influences 
the learning of ICT skills (MCEECDYA, 2010; Nasah, DaCosta, Kinsell, & Seok, 2010). 
Factors that have been shown to be associated include parental socioeconomic status, 
language used at home, ethnicity, and whether or not the student and/or his or her 
parents have an immigrant background. 

A large body of literature shows the influence of students’ socioeconomic background 
on student achievement in a variety of learning areas (see, for example, Saha, 1997; 
Sirin, 2005; Woessmann, 2004). To assess the socioeconomic status of the students’ 
parents, the ICILS student questionnaire includes questions on the highest educational 
levels of parents, their occupations, and the number of books at home. 

In the questionnaire, highest educational levels achieved by the student’s mother 
and father are defined in accordance with the International Standard Classification 
of Education (ISCED) (UNESCO, 2006). The occupation of each parent will be 
recorded through open-ended questions, with occupations classified according to the 
International Standard Classification of Occupations (ISCO) framework (International 
Labour Organisation, 2007) and then scored using the International Socio-economic 
Index (SEI) of occupational status (Ganzeboom, de Graaf, & Treiman, 1992). Home 
literacy resources will be measured through a question asking students to report the 
approximate number of books at home. 

There is evidence from many countries of considerable disparities in students’ access 
to digital resources in homes, and researchers and commentators claim that these 
disparities affect the opportunities that students have to develop the capabilities required 
for living in modern societies (Warschauer & Matuchniak, 2010). ICILS therefore will 
gather information about the digital resources in students’ homes and examine the 
relationship between resource levels and CIL. 

Many studies have found that the cultural and language background of students can 
be associated with their educational performance (see, for example, Elley, 1992; Kao, 
2004; Kao & Thompson, 2003; Stanat & Christinsen, 2006). To measure these aspects 
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of student background, the ICILS student questionnaire includes questions about 
students’ and parents’ country of birth as well as about which language is spoken most 
frequently at home. 

Process-related variables related to the home environment 

Home environment factors that potentially influence the learning process include the 
use of ICT in the home context and learning through interaction with family members. 
The student questionnaire therefore includes questions about the extent to which 
students have learned about different aspects of ICT use from family and/or friends 
and how often they use computers at home in general. 

Individual context 

Antecedent variables at the individual level 

Antecedent variables at the level of the individual student consist of basic background 
characteristics that may influence students’ CIL-related knowledge and skills. Relevant 
factors in this category are age, gender, and educational aspirations. 

Although students’ knowledge and skills generally increase with age, various researchers 
who have collected crossnational data from students in the same grade within an 
education system have found a negative association between age and achievement 
in some countries. However, as Schulz, Ainley, Fraillon, Kerr, and Losito (2010), 
amongst others, explain, the underlying cause for this finding tends to be retention and 
progression policies that lead to the older students in the same grade being those with 
the lower achievement. 

Studies on educational achievement in numerous learning areas have found considerable 
gender-based differences. In particular, crossnational research on reading literacy has 
shown larger gender differences in favor of females (Mullis, Martin, Kennedy, & Foy, 
2007; OECD, 2010b). Males have traditionally tended to be somewhat more proficient 
in mathematics and science, but there is some evidence of a declining gender gap in 
these learning areas (Mullis, Martin, & Foy, 2008; OECD, 2010b). Recent data from two 
cycles of an Australian assessment of ICT literacy in 2008 and 2011 show significantly 
higher levels of achievement for females when compared to male students (Australian 
Curriculum, Assessment, and Reporting Authority [ACARA], 2012; MCEECDYA, 
2008). 

Individual aspirations with regard to education are a further variable that should be 
taken into account during any analysis of variation in students’ CIL. The ICILS student 
questionnaire includes a question that asks students to state which level of educational 
qualification they expect to reach in the future. During analysis of students’ answers to 
this question, categories for this variable will be defined according to the international 
classification of educational qualifications (ISCED) (UNESCO, 2006) and adapted to 
national contexts. 

Process-related variables at the individual level 

These variables consist of attitudinal as well as behavioral factors. Self-beliefs regarding 
proficiency in using ICT are often viewed as central to the process of learning and 
are likely to have a reciprocal association with knowledge and skills as well as with 
the use of ICT applications. It is also important to include student perceptions about 
responsible and appropriate use of ICT. These perceptions can be seen as intended 
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learning outcomes from teaching CIL. Behavioral variables also relate to using ICT for 
different purposes and needs, especially in terms of the potential that frequent and 
varied use of these tools has for facilitating student learning. 

The student questionnaire includes items designed to measure the extent to which 
students express confidence in doing a range of ICT-related tasks. According to 
Bandura (1993), students’ confidence in their ability to carry out specific tasks in an 
area (self-efficacy) is strongly associated with their performance as well as perseverance, 
emotions, and later study or career choices. Moos and Azevedo (2009) concluded from 
their review of research on computer self-efficacy that this variable plays an integral 
role in learning in computer-based learning environments. The two authors examined 
factors related to computer self-efficacy and the relationships between computer self- 
efficacy, learning outcomes, and learning processes. They found a number of positive 
associations between behavioral and psychological factors and computer self-efficacy. A 
particular finding was that students who receive behavioral modeling report significantly 
higher computer self-efficacy than do students taught by more traditional instruction 
methods. 

A related construct is students’ self-concept, which reflects students’ global judgments 
about how they perceive their ability to cope with a certain learning area (Branden, 1994; 
Marsh & Shavelson, 1985). Self-perception has been the basis of several international 
studies of students’ confidence in using ICT (e.g., OECD, 2005). In those studies, overall 
self-confidence was structured around self-confidence in performing routine tasks, 
internet tasks, and high-level tasks. The scales used in the studies indicated substantial 
differences across countries as well as among students in the association between 
students’ self-confidence and the extent and nature of students’ experience of ICT. The 
studies also showed that males tended to express higher levels of self-confidence in 
using ICT than did females. Findings such as these led to items designed to measure 
how students rate their ability to cope with computer technology being included in the 
ICILS student questionnaire. 

Enjoyment of a learning area has the potential to facilitate the acquisition of knowledge 
and skills (Pekrun, Goetz, Titz, & Perry, 2002). Dede, Ketelhut, Clarke, Nelson, and 
Bowman (2005) observed from their study of an ICT-based project which utilized 
graphical multiuser virtual environments that both students and teachers were highly 
engaged, student attendance improved, disruptive behavior dropped, and interesting 
patterns emerged about which students do best under various teaching conditions. The 
ICILS student questionnaire will gather data on students’ enjoyment of ICT learning by 
including a question in which students rate their agreement with statements reflecting 
enjoyment of computing and information technology tasks. 

Applying ICT for different purposes on a regular basis has considerable potential to 
increase knowledge and skills in this area (see, for example, ACARA 2012; Fletcher, 
Schaffhauser, & Levin, 2012). The ICILS student questionnaire consequently includes 
questions about the frequency of using different ICT applications, using the internet for 
social communication, and using ICT for recreational activities. 


Assessment design 


The ICILS test design 

Computer-based testing environment 

The ICILS test is designed to provide students with an authentic computer-based 
assessment experience, balanced with the necessary contextual and functional 
restrictions to ensure that the tests are delivered in a uniform and fair way. In order 
to maximize the authenticity of the assessment experience, the instrument uses a 
combination of purpose-built applications and existing live software. Students need 
to be able to both navigate the mechanics of the test and complete the questions and 
tasks presented to them. Consequently, the test environment comprises two functional 
spaces: the test interface and the stimulus area, as illustrated in Figure 3. 

Figure 3: Test environment comprised of two functional spaces 



Stimulus 

area 


interface 


Test interface 

The test interface serves two purposes. Firstly, it provides information about the test 
(such as test progress, time remaining, and the text for a question or instructions for 
a task). Secondly, it provides navigation controls that allow the participant to move 
between test questions and tasks in much the same way that students would engage 
with other computer- delivered assessments in another discipline. 

Stimulus area 

The stimulus area is a space that contains either noninteractive content, such as an 
image of a login screen for a website, or interactive content, such as electronic texts or 
live software applications. 
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The ICILS test instrument 

The ICILS test instrument consists of questions and tasks that are delivered in 30- 
minute modules. In total, there are four test modules, and each participant completes 
two modules. This rotated module design was chosen to allow for a large number of 
test items (covering the breadth of the CIL framework and a range of difficulties) to be 
included in the instrument without the need for each individual student to complete 
more than two test modules within the allocated time of 60 minutes. 

Test modules 

A test module is a set of linked questions and tasks contextualized by an authentic theme 
and driven by a plausible narrative. Each module has a series of five to eight smaller 
tasks, each of which typically takes students less than one minute to complete, and each 
of which leads up to a single large task. The large tasks typically take 15 to 20 minutes 
to complete. The module themes were selected and tasks developed in accordance with 
the aims of being engaging and relevant to students and of preventing prior content 
knowledge from privileging subgroups of students. 

Table 2 shows the delivery design of the test modules. The design is a fully balanced 
rotated design comprising 12 different module combinations. Module combinations 
are randomly assigned to sampled students so that an approximately equal number of 
students responds to each set within each national sample. The student questionnaire 
takes 20 minutes to complete and is administered after completion of the second test 
module. In order to show the entire assessment sequence for each module combination, 
we have included the student questionnaire in the fourth column of Table 2. 


Table 2: Balanced module rotation for student instruments 


Module 

Combination 

First Module 
(30 minutes) 

Second Module 
(30 minutes) 

Student Questionnaire 
(20 minutes) 

1 

A 

B 

S 

2 

A 

C 

S 

3 

A 

D 

s 

4 

B 

C 

s 

5 

B 

A 

s 

6 

B 

D 

s 

7 

C 

A 

s 

8 

C 

B 

s 

9 

C 

D 

s 

10 

D 

A 

s 

11 

D 

B 

s 

12 

D 

C 

s 


Types of assessment task 

The computer-based assessment of CIL contains three types of task that are integrated 
in a single testing environment. This section contains details of each of these tasks with 
illustrative examples. 4 The examples are taken from a module based on the idea of 


4 At the time of publication of this framework, all ICILS test tasks are secure. The illustrative examples have been created 
for use in this framework to accurately represent the types of task formats and content materials used in ICILS. 
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students working with a group of collaborators to plan the design of a new garden area 
in their school. The ultimate aim of the module is for students to prepare an information 
sheet that explains and engenders support for their garden design in the hope that their 
classmates will vote to have the design used. Creating the information sheet is the large 
task in the module. The contexts and stimulus materials in the example tasks all relate 
to the gardening/garden design theme of the module. 


Task Type 1: Information-based response tasks 

Information-based response tasks use computer technology to deliver pencil-and- 
paper-like questions in a slightly richer form than in paper-based methods. The 
stimulus material is typically a noninteractive representation of a computer-based 
problem or information source. The response formats for these tasks may be multiple- 
choice, constructed-response, or drag-and-drop ones that use the technology only 
to display the stimulus material and record participant responses. In these tasks, the 
computer-based environment is used to capture evidence of students’ knowledge and 
understanding of CIL independently of students using anything beyond the most basic 
skills required to record a response. 

Figures 4 and 5 present two example tasks (shown as Example Tasks 1 and 2) and are 
accompanied with a brief discussion that illustrates the information-based response 
task format. Example Task 1 requires students to examine four organizational-structure 
diagrams for a website and to select the structure that best suits a given set of six pages of 
content. It relates to Aspect 1.3 (managing information) of the CIL construct. Example 
Task 2 requires students to read a noninteractive web-page (in this case, with entries 
from a web-based forum) and to respond using free text in a text entry box in the lower 
(pale green) section of the test interface. 


Figure 4: Example Task 1 (multiple- choice task) 
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Figure 5: Example Task 2 (open-response task) 



The dynamic computer-based environment in Example Task 1 (Figure 4) enables 
students to view each of the four website structures in turn. The stimulus could also 
be presented in a static form (i.e., showing all four diagrams together) in a pencil-and- 
paper test. The simplest multiple-choice tasks in ICILS could also be presented in an 
equivalent form on paper. 

However, because Example Task 1 allows students to drag and drop the web-page 
contents into each organizational-structure template and thereby “try out” the different 
information structures in order to support their choice of the best structure, the 
computer-based stimulus facet of this task extends beyond what could be made easily 
available in a pencil-and-paper format. The task then enables students to provide their 
answer through a conventional multiple-choice format (shown in the light-green lower 
area of the test interface), with one correct response that can be automatically scored. 

While the drag and drop functionality in Example Task 1 serves as an aid to determine 
the correct response, in other ICILS tasks this functionality serves as a method for 
recording student responses. The ICILS assessment uses the drag-and-drop task format 
whenever students are required to classify information into groups or to match objects 
or concepts according to their characteristics. 

The stimulus material in Example Task 2 (reading web-based material) contains 
conflicting information presented as two posts on the forum. The task is presented to 
students as an example of conflicting information and therefore as a vehicle enabling 
them to see that the accuracy of the web-based information may need to be evaluated. 
Example Task 2 relates to Aspect 1.2 (accessing and evaluating information) of the 
CIL construct. Responses to this task are recorded as text fields and scored by scorers 
according to a pre-defined scoring guide. 
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Task Type 2: Skills tasks 

Skills tasks require students to use interactive simulations of generic software or 
universal applications to complete an action. These maybe single-action tasks (such as 
copying, pasting, or opening a web-browser) or may contain a sequence of steps (such 
as “Save As” with a specific file name, or navigation through a menu structure). The 
tasks allow for all possible “correct” responses to be undertaken and are recorded by 
the testing software. All that some skills tasks require students to do is to execute given 
software commands, while others require students to execute commands along with 
some information processing. Skills tasks are scored automatically. 

The ICILS student test contains linear and nonlinear skills tasks. A linear skills task may 
be as simple as executing a single command (such as opening a file from the desktop), 
or requiring more than one step to complete the task. All appropriate methods of 
executing a command (e.g., using the mouse, pull-down menus, or keyboard shortcuts) 
are scored as equivalent and correct. Linear skills tasks that require the execution of 
more than one command can only be completed correctly if the commands are executed 
in a necessary prescribed sequence. For example, if students are instructed to copy and 
paste an image, they would first need to select the image and then execute the copy and 
paste commands in that order. Responses are automatically detected and scored once 
participants have reached an “endpoint” to a task. Figure 6, containing Example Task 3, 
illustrates a linear skills task. 

Figure 6: Example Task 3 (linear skills task) 
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Example Task 3 requires students to change the settings for a document in a collaborative 
workspace in order to restrict viewing access to specified people. Students must first 
click on the settings/sharing menu link and then make changes within a dialogue 
box to restrict the file sharing to their group. Example Task 3 relates to Aspect 1.1 
(understanding computer use) of the CIL construct. 

Nonlinear skills tasks require students to execute a software command (or reach a desired 
outcome) by executing subcommands in a number of different sequences. Example Task 
4, presented in Figure 7, illustrates a nonlinear skills task. This task requires students to 
use the filtering functions of a web-based database and to interpret some simple text in 
order to locate an object (a plant) that matches a given set of characteristics. The task 
is thus an example of a nonlinear skills task that requires information-processing skills 
and relates to Aspect 1.3 (managing information) of the CIL construct. The web-based 
database contains too many objects for a student to search manually with ease. As such, 
the automatic scoring gives the highest level of credit to students who make use of the 
filtering functions (in any order) to support their search. Students who identify the 
correct task without using the filters receive less credit. 

Figure 7: Example Task 4 (nonlinear skills task) 
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Task Type 3: Authoring tasks 

Authoring tasks require students to modify and create information products using 
authentic computer software applications. The applications, purpose-built for ICILS, 
adhere to software application conventions, such as the use of standard icons, or typical 
response types to given commands. This approach may require students to use multiple 
applications concurrently (such as email applications, web-pages, spreadsheets, and 
wordprocessing or multimedia software) as one typically does when using computer 
software to perform authentic, complex tasks. Each student’s work is automatically 
saved as an information product file for subsequent assessment by scorers according 
to a prescribed set of criteria. Example Task 5 (Figure 8) illustrates a simple authoring 
task. 

Figure 8: Example Task 5 (simple authoring task) 



Example Task 5 requires students to use some simple map-drawing software to create 
a garden design plan that represents the text describing the plan. It relates to Aspect 
2.1 (transforming information) of the CIL construct. The task is a simple authoring 
task because it asks students to use only the instructions and one piece of software (the 
mapping software) to complete the task. It is also simple because there is a relatively 
narrow range of “correct” ways in which the student can draw the garden design to 
match the text specifications. The task is manually scored according to the accuracy with 
which the different specified elements of the garden design are shown in the diagram. 

The complex authoring task (Task 6) illustrated in Figure 9 requires students to use 
information from a range of electronic sources to create an information sheet that 
explains and promotes their garden design. The stimulus is nonlinear, fully interactive, 
and behaves intuitively. Students can tab between concurrent applications such as 
the web-browser and presentation software. They can copy and paste text and images 
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between applications and can use the software to design the information sheet. The 
final information product is saved, stored, and then scored against a set of criteria. 
The scoring criteria can be categorized as relating to students’ use of (1) the software 
features and (2) the available information. 

Figure 9: Example Task 6 (complex authoring task) 



Criteria relating to students’ use of software features can include ability to use color, text 
formatting, and general page layout. These criteria typically have an internal hierarchy 
based on the degree to which the software features are used to support or even enhance 
the communicative effect of the information product. Criteria relating to students’ use 
of information can also include students’ adaptation of information, the relevance (and 
accuracy) of information selected for and used in the information product, and the 
appropriateness of selected information for the target audience. Note, though, that the 
use of information is only assessed with respect to students’ use of the information 
provided to them for use in the module. 

The highest level of credit is given to student work that demonstrates ability to use 
the software features to enhance the communicative effect of the information product. 
The lowest level of credit is given to pieces of work that show no application of the 
relevant software feature, or uncontrolled use (such as extremely poor color contrast 
or overlapping text) that inhibits comprehension of the product. The range of criteria 
available to evaluate Example Task 6 means that the single task allows for students 
providing evidence of achievement relating to Aspects 2.1 (transforming information), 
2.2 (creating information), and 2.3 (sharing information) of the CIL construct. 




ASSESSMENT DESIGN 


43 


Mapping test items to the CIL framework 

The test items that comprise the assessment modules are based on the strands and 
constituent aspects in the assessment framework described earlier. The CIL framework 
is central to the process of instrument development because it provides a theoretical 
underpinning for the assessment and a means of describing its content. Table 3 shows 
the mapping of the test items to the two assessment strands, the constituent aspects, 
and their levels in the CIL framework. 


Table 3: Mapping test items to the CIL framework 


CIL Aspect 

Total 

(Items) 

Total 

(Score Points) 

Strand 1 : Collecting and managing information 



Aspect 1.1: Knowing about and understanding computer use 

12 

12 

Aspect 1 .2: Accessing and evaluating information 

10 

14 

Aspect 1 .3: Managing information 

5 

6 

Total (Strand 1) 

27 

32 

Strand 2: Producing and exchanging information 



Aspect 2.1 : Transforming information 

16 

23 

Aspect 2.2: Creating information 

15 

23 

Aspect 2.3: Sharing information 

8 

12 

Aspect 2.4: Using information securely and safely 

14 

15 

Total (Strand 2) 

53 

73 


We can see from Table 3 that about twice as many items and score points relate to 
Strand 2 than relate to Strand 1 of the CIL construct. The main reason for this is that 
the large tasks at the end of each module focus on students’ creation of an information 
product and therefore require each of these tasks to be assessed via multiple criteria 
with multiple score categories. Assessment of the large tasks focuses on Aspects 2.1 and 
2.2, and together these contribute the largest number of associated score points across 
the four test modules. 

The test design of ICILS was not planned to assess equal proportions of all aspects 
of the CIL construct, but rather to ensure some coverage of all aspects as part of an 
authentic set of assessment activities in context. The balance of items and score points 
relating to the different aspects of the CIL construct reflects the balance of time that 
students are expected to spend completing the different tasks. 
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The ICILS student questionnaire and context instruments 

Student questionnaire 

The student questionnaire has been designed primarily to collect data that address 
Research Questions 3 and 4. 

• What characteristics of students’ levels of access to, familiarity with, and self- 
reported proficiency in using computers are related to student achievement in 
computer and information literacy? 

a) How do these characteristics differ among and within countries? 

b) To what extent do the strengths of the relations between these characteristics 
and measured computer and information literacy differ among countries? 

• What aspects of students’ personal and social backgrounds (such as gender, 
socioeconomic background, and language background) are related to computer 
and information literacy? 


ICILS will use the data gathered from the student questionnaire for two purposes. 
Firstly, these data will be used in analyses that examine the relationships between 
student-level factors and measured CIL. Secondly, these data will be used to provide 
descriptive information about patterns of computer access and use across and within 
countries. 

The student questionnaire is designed to collect indices of student and home background, 
namely: 

• Age (in years); 

• Gender; 

• Expected highest level of educational qualifications; 

• Immigration background; 

• Language used at home (test language or others); 

• Mother’s and father’s highest occupational status; 

• Mother’s and father’s highest level of education; 

• Home literacy (number of books at home); 

• ICT resources at home; and 

• Experience with ICT. 

The student questionnaire thus contains questions that will derive indices on students’ 
ICT use and students’ attitudes toward ICT: 

• Use of computer applications (frequency); 

• Use of ICT for social communication (frequency); 

• Use of ICT for exchanging information (frequency); 

• Use of ICT for recreational purposes (frequency); 

• Use of ICT for school-related purposes (frequency); 

• Use of ICT in school subject lessons (frequency); 

• Reports on learning ICT tasks at school; 
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• ICT self-efficacy; 

• Interest and enjoyment of computer use; and 

• Self-concept as related to computer use. 

Teacher questionnaire 

This questionnaire is concerned with information about teachers’ perceptions of ICT 
in schools and the use that teachers make of ICT in educational activities during their 
teaching. Together with the questionnaires completed by the school principal and the 
ICT coordinator, this questionnaire will collect data that address Research Question 2: 

What aspects of schools and education systems are related to student achievement 
in CIL? 


The assumption underlying this question is that the extent to which and the ways in 
which ICT is used in schools influence the development of students’ CIL. Information 
from the teacher questionnaire will also be used to describe the use of ICT in pedagogy 
within and across countries and within and across major teaching areas. It will not be 
possible, within the scope of ICILS, to link teacher-based information to individual 
students. Rather, this information will be used to generate school-level indicators that 
will be used, along with student-based data, in two-level multiple regression analyses. 

The population for the ICILS teacher survey is defined as all teachers teaching regular 
school subjects to the students in the target grade (generally Grade 8) at each sampled 
school. Fifteen teachers will be selected at random from all teachers teaching the target 
grade at each sampled school. 5 This cluster size is required in order to produce: 

• School-level estimates with sufficient precision to be used in analyses that examine 
associations with student outcomes; and 

• Population estimates with a precision level similar to the levels generated from the 
student data. 

The teacher questionnaire consists of questions regarding teachers’ background, 
teachers’ familiarity with ICT, their use of ICT when teaching a reference class, and 
their perceptions of ICT provision and use at school. Teachers will also be asked about 
their experience of learning to use ICT in teaching. 

The teacher questionnaire is therefore designed to generate the following indices about 
teachers’: 

• CIL self-efficacy; 

• Use of ICT tools in class; 

• Reports on student engagement in learning activities with ICT; 

• Reports on use of ICT in teaching and learning practices; 

• Perceptions of emphasis on ICT skills development in class; 

• Positive views regarding the use of ICT in teaching and learning; 

• Negative views regarding the use of ICT in teaching and learning; 

• Perceptions of impediments to ICT use at school; 


5 In small schools, this will mean all teachers of Grade 8 students. 
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• Participation in ICT professional development; and 

• Perceptions of collaborations involving ICT use. 

School questionnaires 

ICILS has two school questionnaires, each of which complements the other: a principal 
questionnaire and an ICT coordinator questionnaire. While ICILS intends each 
questionnaire to be completed by different people in each school, ICILS does provide 
for the possibility that both may be completed by the same person in a small school 
where there is no identifiable ICT coordinator. 

Principal questionnaire 

The principal questionnaire is designed primarily to collect data that address Research 
Question 2: 

What aspects of schools and education systems are related to student achievement 
in CIL? 


The assumption underlying this research question is that the extent to which and the 
manner in which ICT is used in schools influence the development of students’ CIL. 
It is expected that principals will provide important perspectives on school practices 
and policies regarding the pedagogical use of ICT. Another purpose of the principal 
questionnaire is to collect data that will contribute to a description of the contexts 
within which ICT is used for pedagogical purposes both in and across the countries 
participating in ICILS. The ICILS principal questionnaire therefore covers these areas: 

• Characteristics of the principal, including his or her use of ICT; 

• School characteristics (number of enrolments, range of grades taught, characteristics 
of the school location, ratio of female to male enrolments); 

• Management of ICT in the school; 

• Encouragement of ICT use in teaching and learning; 

• The school’s pedagogical orientations; and 

• Provision within the school for professional development in “operating” ICT tools 
and using ICT for pedagogical purposes. 

The questionnaire items are therefore designed to generate the following indices: 

• School principals’ use of computers for school-related purposes (frequency); 

• School size (student enrolment); 

• Student-teacher ratio; 

• School management (public/private); 

• School principals’ perceptions of the importance of ICT use at school; 

• School principals’ reports on expectations of teachers’ ICT skills; 

• School principals’ reports on ICT policies and procedures; 

• School principals’ reports on teachers’ professional development for ICT use; and 

• School principals’ reports on school priorities for ICT use in teaching and learning. 
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ICT coordinator questionnaire 

The ICT coordinator questionnaire has been designed primarily to collect data that 
address Research Question 2: 

What aspects of schools and education systems are related to student achievement 
in CIL? 


The assumption underlying Research Question 2 is that the extent to which and the 
manner in which ICT is used in schools influence the development of students’ CIL. It 
is anticipated that the ICT coordinator will provide important perspectives on school 
practices and policies regarding the pedagogical use of ICT. An additional purpose for 
the questionnaire is to provide data that will contribute to a description of the contexts 
within which ICT is used for pedagogy within and across countries. 

The ICT coordinator questionnaire will therefore collect data on the following: 

• ICT resources (numbers of computers of different types, availability of computers 
for student use, availability of other ICT devices, availability of digital learning 
resources, and networking and internet connectivity); 

• ICT use in the school (provision of specialist teaching in ICT, emphasis in curriculum 
areas, learning management systems, and school administration); and 

• ICT technical support (maintenance provision and support for managing resources); 
and 

• Provision for ICT-related professional development at school. 

The questions in the ICT coordinator questionnaire are therefore designed to generate 
two indices: 

• The computer-student ratio in the participating schools; and 

• The quality of ICT resources in these schools. 

National contexts survey 

The intention behind the national contexts survey is to collect data that address Research 
Question 2: 

What aspects of schools and education systems are related to student achievement 
in CIL? 


The assumption underlying this question is that the opportunities students have to use 
ICT will have an impact on their opportunities to learn about CIL and therefore on the 
development of their CIL. 

Data from the national contexts survey will be used to generate CIL-education profiles 
in the participating countries. The survey is also designed to provide data on contextual 
factors, such as the structure of the education system and education policy. This 
information will facilitate the analysis of differences in CIL education across countries. 
More specifically, data from the national contexts survey will be used for three broad 
purposes: 

• To generate systematic descriptions of ICT-related policy and practice in school 
education across the ICILS countries; 
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• To provide the bases for system-level analyses of differences in CIL across the ICILS 
countries; and 

• To provide systematic data that researchers and other stakeholders can use as a basis 
for interpreting crossnational differences in the patterns of relationships among 
factors related to CIL achievement. 

Finally, information obtained from the national contexts survey will also be used to 

supplement the following types of data pertaining to education systems: 

• General approaches to the use of ICT in schools; 

• Approaches to the development of CIL in schools; 

• Availability of and access to ICT resources in schools; and 

• The extent to which ICT in teaching and learning is encouraged. 


APPENDIX A: 

Organizations and individuals involved in ICILS 

International study center 

The international study center is located at the Australian Council for Educational 
Research (ACER) and serves as the international study center for ICILS. Center staff at 
ACER were responsible for designing and implementing the study in close cooperation 
with the IEA Data Processing and Research Center (DPC) in Hamburg, Germany, and 
the IEA secretariat in Amsterdam, the Netherlands. 

Staff at ACER 

Julian Fraillon, research director 
John Ainley, project coordinator 
Wolfram Schulz, assessment coordinator 
Tim Friedman, project researcher 
Daniel Duckworth, test development 
Karin Hohlfeld, test development 
Eveline Gebhardt, data analyst 

International Association for the Evaluation of Educational 
Achievement (IEA) 

IEA provides overall support in coordinating ICILS. The IEA secretariat in Amsterdam, 
the Netherlands, is responsible for membership, translation verification, and quality 
control monitoring. The IEA Data Processing and Research Center (DPC) in Hamburg, 
Germany, is mainly responsible for sampling procedures and the processing of ICILS 
data. 

Staff at the IEA Secretariat 

Hans Wagemaker, executive director 

Paulina Korsnakova, senior professional officer and researcher 
Barbara Malak, manager membership relations 
David Ebbs, research officer (translation verification) 

Alana Yu, publications officer 
Jur Hartenberg , financial manager 
Isabelle Gemin, financial assistant 

Staff at the IEA Data Processing and Research Center (DPC) 

Dirk Hastedt, co-director 
Ralph Carstens, co-project manager 
Michael Jung, co-project manager 
Sabine Meinck, researcher (sampling) 


50 


ICILS ASSESSMENT FRAMEWORK 


SoNET Systems 

SoNET Systems was responsible for developing the software systems underpinning the 
computer-based student assessment instruments. This work included development of 
the test and questionnaire items, the assessment delivery system, and the web-based 
translation, scoring, and data-management modules. 

Staff at SoNET Systems 

Mike Janie, managing director 

Stephen Birchall, general manager of software development 
Erhan Halil, senior analyst programmer 
Rakshit Shingala, analyst programmer 
Stephen Ainley, quality assurance 
Ranil Weerasinghe, quality assurance 

ICILS Project Advisory Committee (PAC) 

PAC has, from the beginning of the project, advised the international study center and 
its partner institutions during regular meetings. 

PAC members 

John Ainley (chair), ACER, Australia 
Ola Erstad, University of Oslo, Norway 
Kathleen Scalise, University of Oregon, US 
Alfons ten Brummelhuis, Kennisnet, the Netherlands 

ICILS sampling referee 

Jean Dumais from Statistics Canada in Ottawa is the sampling referee for the study. He 
has provided invaluable advice on all sampling-related aspects of the study. 

National research coordinators 

The national research coordinators (NRCs) played a crucial role in the development of 
the project. They provided policy- and content-oriented advice on the development of 
the instruments and are responsible for the implementation of ICILS in the participating 
countries. 

Australia 

Lisa DeBortoli 

Australian Council for Educational Research (ACER) 

Buenos Aires (Argentina) 

Silvia Montoya 

Assessment and Accountability, Ministry of Education 

Canada 

Melanie Labrecque 

Council of Ministers of Education (CMEC) 

Chile 

Gabriela Cares 

Division de Estudios, Agenda de Calidad de la Education 
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Croatia 

Michelle Bras Roth 

National Centre for External Evaluation of Education 

Czech Republic 

Josef Basl 

Czech School Inspectorate 

Denmark 

Jeppe Bundsgaard 

Institutfor Uddannelse og Pcedagogik, Aarhus Universitet 

Germany 

Wilfried Bos 

Institute for School Development Research, TU Dortmund University 
Birgit Eickelmann 

Institute for Educational Science, University of Paderborn 

Hong Kong, SAR 

Nancy Law 

Centre for Information Technology in Education, the University of Hong Kong 

Korea, Republic of 

Soojin Kim 

Korea Institute for Curriculum and Evaluation 

Lithuania 

Eugenijus Kurilovas 
Asta Buineviciute 

Centre of Information Technologies in Education 

Netherlands 

Martina Meelissen 

Department of Research Methodology, Measurement and Data Analysis, University of 
Twente 

Alfons ten Brummelhuis 
Kennisnet 

Norway 

Inger Throndsen 

Department of Teacher Education and School Research, University of Oslo 
Geir Ottestad 

Norwegian Center for ICT in Education 

Poland 

Kamil Sijko 

The Educational Research Institute (IBE) 

Russian Federation 

Svetlana Avdeeva 

Center for International Co-operation in Education Development 


52 


ICILS ASSESSMENT FRAMEWORK 


Slovak Republic 

Andrea Galadova 

National Institute for Certified Educational Measurements (NUCEM) 

Slovenia 

Eva Klemencic 
Barbara Brecko (field trial) 

Center for Applied Epistemology, Educational Research Institute 

Switzerland 

Per Bergamin 

Fernfachhochschule Schweiz 

Thailand 

Chaiwuti Lertwanasiriwan 

Institute for the Promotion of Teaching Science and Technology (IPST) 

Turkey 

Giil<;in Oz 

Meral Alkan (field trial) 

Ministry of National Education, General Directorate of Innovation and Educational 
Technologies 

The following countries participated in the field trial, and the following people acted 
in the role of NRC until the field trial, after which their countries withdrew from the 
study. 

Israel 

Boaz Rozenbaum and Inbal Ron Kaplan (field trial) 

National Authority for Measurement and Evaluation in Education (RAMA ) 

Spain 

Jesus Dominguez Castillo (field trial) 

National Institute for Evaluation of Education 
David Cervera Olivares (field trial) 

National Institute for Evaluation of Education 
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Hypothetical progress map for CIL 


The ICILS research team developed the following hypothetical progress map for CIL in 
order to achieve the following: 

1 ) Support readers’ understanding of the nature of the contents of the CIL construct 
operationalized through the ICILS student test; and 

2) Support readers’ understanding of how progress in CIL achievement has been 
conceptualized for the purpose of framing ICILS and developing the CIL test 
instrument. 

The levels and broad nature of progress described in the progress map are based on 
an empirically derived map of ICT literacy progress established using data from over 
6,000 Grade 6 and Grade 10 students who completed module-based assessment tasks 
underpinned by a similar assessment design to that of ICILS (see Australian Curriculum, 
Assessment, and Reporting Authority, 2012). The contents of the progress map have 
been adapted to represent the two strands and to include examples of task types that 
reflect the aspects of the CIL construct. 

The empirical student achievement data, together with the contents of the ICILS student 
test, are used here in order to describe the empirically based CIL achievement scale that 
will be used after the ICILS data collection in 2013. 
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Hypothetical progress map for CIL 


Level Strand 1 : Collecting and managing 
information 


5 Students working at Level 5 evaluate 
the credibility of information from 
electronic sources and select the most 
relevant information to use for specific 
communicative purposes. They create 
structures for simple databases and 
file-management systems and evaluate 
the efficacy of simple data structures. 

Thus, students working at Level 5 can, for 
example: 

• Select and include information from 
electronic resources in an information 
product to suit an explicit communicative 
purpose; 

• Explain how the features of a web-based 
text such as hyperbole or extreme bias 
contribute to the credibility of the text; 

• Identify some of the frequent “hooks” 
used in phishing and other common 
web-based deceptions such as 
unsubstantiated requests for information, 
or extraordinary claims or offers; 

• Apply specialized software and file- 
management functions such as using 
the history function on a web-browser 
to return to a previously visited page or 
sorting data in a spreadsheet according 
to a specified criterion; 

• Specify field parameters that can be used 
to organize data relevant to their content; 

• Suggest ways an existing database or file 
system could be reorganized to make it 
more efficient. 


Strand 2: Producing and exchanging 
information 


Students working at Level 5 create 
information products that show evidence 
of planning and technical competence. 

They use software features to reshape 
and present information consistent with 
presentation conventions. They design 
information products that combine different 
elements and accurately represent their 
source data. They use available software 
features to enhance the appearance of 
their information products. Students show 
awareness of the power of information and 
the contexts in which information sharing 
can be socially constructive or destructive. 
Thus, students working at Level 5 can, for 
example: 

• Create an information product in which 
the information flow is clear and logical 
and the tone and style are consistent and 
appropriate for a specified audience; 

• Use graphics and text-software-editing 
features such as font formats, color, 
and animations consistently within an 
information product to suit a specified 
audience; 

• Create tables and charts that accurately 
represent data and include them in an 
information product with text that refers 
to their contents; 

• Give examples of social contexts in which 
information can be used to disseminate 
socially significant information; 

• Explain how communication 
networks can be used to promulgate 
misinformation and suggest ways of 
protecting against these actions. 
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Hypothetical progress map for CIL (contd.) 


Level Strand 1 : Collecting and managing 
information 


4 Students working at Level 4 generate well- 
targeted searches for electronic information 
sources, select relevant information from 
within sources to meet a specific purpose, 
and suggest strategies for checking the 
veracity of information sources. They 
recognize and make use of metadata 
when retrieving and managing files. Thus, 
students working at Level 4 can, for 
example: 

• Independently select and use appropriate 
software and/or hardware to suit specific 
tasks, purposes, and social contexts; 

• Independently modify the settings for an 
individual task using a peripheral device, 
such as a printer that can print on both 
sides of a page; 

• Suggest ways that the veracity of web- 
based information can be confirmed; 

• Use fields that provide the identifying 
characteristics of data needed to search, 
sort, and retrieve information from within 
a database (such as an electronic media 
manager or a web-based catalogue); 

• Identify the features/uses of common file 
types according to their extensions (such 
as .doc, .xls, .gif.); 

• Generate searches that target relevant 
resources for a specified purpose; 

• Select sections relevant to a given 
purpose from within electronic resources. 


Strand 2: Producing and exchanging 
information 


Students working at Level 4 create 
information products with simple linear 
structures and use software commands to 
edit and reformat information products in 
ways that demonstrate some consideration 
of audience and communicative purpose. 
They create information products in which 
the flow of information is clear and the tone 
is controlled to suit a specified audience. 
They recognize that shared information can 
be tailored to suit and can have different 
effects on different audiences. They also 
recognize that there are risks associated 
with sharing information with others, and 
can suggest ways of minimizing these risks. 
Thus, students working at Level 4 can, for 
example: 

• Select and apply graphics and text- 
software-editing features such as font 
formats, color, and image placement 
consistently across a simple information 
product; 

• Combine mixed-media resources such as 
graphics, text, audio, and video; 

• Use software to draw graphs of tables of 
data to demonstrate patterns; 

• Create a flow chart to represent a 
decisionmaking system; 

• Identify security risks associated with 
internet data and explain the importance 
of respecting and protecting the 
intellectual property rights of authors; 

• Suggest ways of using software to 
present a given set of information for 
different audiences; 

• Suggest the different potential size and 
breadth of an audience for information 
presented using different electronic 
communication systems; 

• Identify ways of minimizing undesirable 
access or use of electronically shared 
information, and use software options 
and parameters to restrict access or limit 
use. 


56 


ICILS ASSESSMENT FRAMEWORK 


Hypothetical progress map for CIL (contd.) 


Level 


Strand 1 : Collecting and managing 
information 


Strand 2: Producing and exchanging 
information 


3 


Students working at Level 3 demonstrate 
some autonomy when using computers 
as information-gathering and management 
tools. They generate simple general search 
questions and select the best information 
source to meet a specific purpose. They 
retrieve information from given electronic 
sources to answer specific concrete 
questions and manage files effectively 
within simple organizational structures. 

Thus, students working at Level 3 can, for 
example: 

• Recognize the role of the server and 
clients on a computer network; 

• Retrieve information from a database 
such as a library catalogue; 

• Recognize the purpose of including 
usernames and passwords to access files 
on shared networks; 

• Create a meaningful organizational 
system for a set of files based on their 
type and/or content; 

• Recognize the key features of an 
“operating system’’; 

• Recognize the difference between the 
“save” and “save as” commands; 

• Identify that two different search terms 
relating to the same topic can result in 
different numbers of “matches” on a 
search engine. 


Students working at Level 3 assemble 
information in a provided simple linear 
order to create information products. They 
follow instructions to use conventionally 
recognized software commands to edit 
and reformat information products. They 
recognize that communication with ICT 
has responsibilities for users and offers the 
potential for misuse. Thus, students working 
at Level 3 can, for example: 

• Use graphics and text-software-editing 
features to manipulate aspects such 
as color, image size, and placement in 
simple information products; 

• Apply templates or styles, when 
instructed, to improve the appearance 
and layout of documents and text; 

• Assemble a liner sequence of video clips 
with simple transitions; 

• Apply simple animations to objects to 
demonstrate a process or dynamic 
action; 

• Suggest different contexts in which 
different electronic communications 
systems may be most appropriate; 

• Identify some of the responsibilities 
of contributors to collaborative online 
projects or information resources such as 
wikis and review sites; 

• Recognize the potential for ICT 
misuse through information sharing 
and communications networks such 
as plagiarism and deliberate identity 
concealment as well as suggest 
measures to protect against them. 
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Hypothetical progress map for CIL (contd.) 


Level 


2 


Strand 1 : Collecting and managing 
information 


Students working at Level 2 use computers 
as tools to complete very basic and explicit 
information-gathering and management 
tasks. They locate simple, explicit 
information from within a given electronic 
source, recognize common computer 
conventions, and demonstrate basic 
knowledge of how computers function as 
tools. Thus, students working at Level 2 
can, for example: 

• Recognize that file extensions such as .txt 
or .gif represent the type of information 
stored in a file; 

• Add a web-page to a list of favorites 
(bookmarks) in a web-browser; 

• Recognize that computers “run” 
programs that can be used to complete a 
range of functions; 

• Click on buttons in a web-page that have 
links to explicitly stated information; 

• Recognize that information in a working 
document can only be retrieved if the file 
is “saved;” 

• Recognize that individual files must each 
have a different name when saved to the 
same location in a directory tree; 

• Move a file from one folder to another in a 
simple directory tree; 

• Select the most relevant search term from 
a set of possible terms. 


Strand 2: Producing and exchanging 
information 


Students working at Level 2 use 
computers to add content to and make 
simple changes to existing information 
products when instructed. They edit 
information products and create products 
that show limited consistency of design 
and information management. Students 
identify the efficiency of immediate 
communication with multiple parties using 
communications software and recognize 
common communications conventions. 
Thus, students working at Level 2 can, for 
example: 

• Make changes to some presentation 
elements in an information product; 

• Apply simple software reformatting 
functions such as copying and pasting 
information across columns in a 
spreadsheet; 

• Use a drawing tool to copy and repeat 
design elements and create patterns; 

• Send emails to groups of users or 
establish “friends” on a social networking 
site; 

• Recognize differences between the 
To, Cc and BCc functions in email, or 
different classifications of “friends” on 
social networking software; 

• Recognize appropriate email greetings 
and sign-offs when communicating with 
different people. 
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Hypothetical progress map for CIL (contd.) 


Level 


1 


Strand 1 : Collecting and managing 
information 


Students working at Level 1 demonstrate a 
functional working knowledge of computers 
as tools to complete tasks. They implement 
the most commonly used file-management 
and software commands when instructed. 
They recognize the most commonly used 
ICT terminology and functions. Thus, 
students working at Level 1 can, for 
example: 

• Apply basic file and computer 
management functions such as opening 
and dragging and dropping files on the 
desktop; 

• Apply generic software commands such 
as the "save as’’ and “paste’’ function or 
select all the text on a page; 

• Recognize basic computer use 
conventions such as identifying the main 
parts of a computer and that the “shut- 
down’’ command is a safe way to turn off 
a computer; 

• Recognize different types of commonly 
used software such as wordprocessors, 
internet search engines, and web- 
browsers; 

• Recognize the function of some computer 
peripheral devices such as USB drives, 
DVD drivers, and printers. 


Strand 2: Producing and exchanging 
information 


Students working at Level 1 perform basic 
communication tasks using computers and 
software. They recognize different software 
communications systems and can compile 
text and messages using the most basic 
features of these systems. Thus, students 
working at Level 1 can, for example: 

• Apply graphics-manipulation-software 
features such as adding and moving 
predefined shapes to reproduce the basic 
attributes of a simple image; 

• Apply commonly used text formatting 
commands such as “bold” or “italic” to 
modify the appearance of fonts; 

• Recognize the difference between 
communication systems such as email, 
instant messaging, blogs, and social 
networking software; 

• Prepare an email by inserting an address 
and subject; 

• Identify that the appearance and layout 
of text and graphics can influence the 
communicative efficacy of an electronic 
text. 
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This document outlines the framework and assessment design for 
the International Computer and Information Literacy Study (ICILS) 
sponsored by the International Association for the Evaluation of 
Educational Achievement (IEA). Over the past 50 years, IEA has 
conducted comparative research studies focusing on educational 
policies, practices, and outcomes in more than 80 countries. 

Information technologies are a pervasive feature of modern societies. 
Knowing about, and knowing how to use, these technologies have 
become essentials of the modern era. The purpose of ICILS is to 
investigate, in a range of countries, the ways in which young people 
are developing computer and information literacy (CIL) to support 
their capacity to function in the digital age. This document defines 
the knowledge, understanding, and skills that make up CIL, postulates 
aspects of the contexts in which CIL develops in young people, and 
embeds these in a conceptual framework which underpins the ICILS 
study design. This document also outlines innovative and authentic 
computer-based assessment methods which are used in the study. 
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